Download pictures (or videos) along with their captions and other metadata from Instagram.
- instagram-client
- instagram-feed
- downloader
Repository Index
47,133 repositories indexed
25–48 of 117 · Most stars
Incredibly fast crawler designed for OSINT.
Browser automation and testing without limits. SeleniumBase's CDP Mode bypasses bot-detection with ease. Check out the examples!
- python
- selenium-python
- e2e-testing
- seleniumbase
TranslatedXiaohongshu (XiaoHongShu, RedNote) link extraction / post collection tool
TranslatedAn online tool for batch downloading WeChat Official Account articles. It supports exporting read count and comment data, requires no environment setup, and can be used through an online website. It supports private deployment with docker and deployment on Cloudflare. It supports downloading various file formats, and the HTML format can restore article layout and styles 100%.
- download
- wechat-article
- wechat-download
TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.
- webcrawler
- scrapy
- crawlab
- spiders-management
Convert any URL to an LLM-friendly input with a simple prefix https://r.jina.ai/
A scalable web crawler framework for Java.
A standalone version of the readability lib
jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety.
TranslatedA new-generation web crawler platform that defines crawler workflows graphically, so you can build a crawler without writing code.
TranslatedAV movie management system, avmoo, javbus, javlibrary crawler, online AV movie library, AV magnet link database, Japanese Adult Video Library, Adult Video Magnet Links - Japanese Adult Video Database
- javbus
- avmoo
- javlibrary
- spider
The RSS feed for websites missing it
- php
- rss-bridge
- unlicense
- atom-feed
Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.
- python
- machine-learning
- natural-language-processing
- web-mining
A Chrome DevTools Protocol driver for web automation and scraping.
- cdp
- chrome-headless
- chrome-devtools
- chrome-devtools-protocol
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
- web-scraping
- text-extraction
- nlp
- html2text
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
The next web scraper. See through the <html> noise.
Dark Web OSINT Tool
- tor
- tor-network
- deepweb
- dark-web
SwiftSoup: Pure Swift HTML Parser, with best of DOM, CSS, and jquery (Supports Linux, iOS, Mac, tvOS, watchOS)
- swift
- swiftsoup
- parse
- html-document
Open Source Web Crawler for Java
Mechanize is a ruby library that makes automated web interaction easy.
- scraping
- web
- ruby
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
- crawler
- dotnetcore
- cross-platform
- csharp
A browser testing and web crawling library for PHP and Symfony