A Smart, Automatic, Fast and Lightweight Web Scraper for Python
Repository Index
55,640 visible repositories in the index
25–48 of 81 · Most stars
TranslatedPython API for JMComic | Provides a Python API for accessing 禁漫天堂 (JMComic), supporting both the web and mobile clients | 禁漫天堂 (JMComic) GitHub Actions downloader🚀
- 18comic
- crawler
- github-actions
- pypi
Pholcus is a distributed high-concurrency crawler software written in pure golang
- spider
- crowler
The AI local media metadata crawler
- emby
- kodi
- plex
- python
One-stop Proxies Crawling and Aggregation Platform
- proxypool
A Chrome DevTools Protocol driver for web automation and scraping.
- cdp
- chrome-headless
- chrome-devtools
- chrome-devtools-protocol
🔥 The fastest and powerful Python library for Instagram Private API 2026 with HikerAPI SaaS
- instagram-api
- instagram-client
- instagram-downloader
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
- web-scraping
- text-extraction
- nlp
- html2text
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
TranslatedA WeChat official account crawler interface based on Sogou WeChat search
The All in One Framework to Build Undefeatable Scrapers
- anti-bot
- anti-detection
- cloudflare-bypass
- cloudflare-scrape
TranslatedHands-on 🐍 web crawlers for many kinds of websites and e-commerce data 🕷. Includes 🕸: Taobao products, WeChat official accounts, Dianping, Qichacha, job recruitment sites, Xianyu, Alibaba tasks, Cnblogs, Weibo, Baidu Tieba, Douban movies, Baotu (ibaotu.com), Quanjing (quanjing.com), Douban music, a provincial drug administration, Sohu News, machine learning text collection, FOFA asset collection, Autohome, National Bureau of Statistics, Baidu keyword indexed-page counts, spider pool directories, Toutiao (Today's Headlines), Douban movie reviews, Ctrip, Xiaomi App Store, Anjuke, Tujia homestays ❤️❤️❤️. WeChat crawler showcase project:
- python3
- crawler
- baidu-tieba
- taobao-spider
Redis-based components for Scrapy.
- scrapy
- crawler
- distributed
- redis
Distributed crawler powered by Headless Chrome
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
- high-availability
- scrapy
- ipproxy
- distributed
Dark Web OSINT Tool
- tor
- tor-network
- deepweb
- dark-web
Open Source Web Crawler for Java
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
- crawler
- dotnetcore
- cross-platform
- csharp
Headless Chrome .NET API
- puppeteer
- chrome
- chromium
- automation
TranslatedMulti-platform content monitoring, collection, and reposting: one web panel to manage Douyin / Xiaohongshu / Kuaishou / WeChat Channels / TikTok
- automation
- content-monitoring
- crawler
- douyin
A browser testing and web crawling library for PHP and Symfony
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
Find web directories without bruteforce
- pentesting
- security
- security-tools
- websec