Repository Index

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

• scraping
• scraper
• scrape
• webscraping

TranslatedPython API for JMComic | Provides a Python API for accessing 禁漫天堂 (JMComic), supporting both the web and mobile clients | 禁漫天堂 (JMComic) GitHub Actions downloader🚀

• 18comic
• crawler
• github-actions
• pypi

Pholcus is a distributed high-concurrency crawler software written in pure golang

• spider
• crowler

One-stop Proxies Crawling and Aggregation Platform

• proxypool

The AI local media metadata crawler

• emby
• kodi
• plex
• python

A Chrome DevTools Protocol driver for web automation and scraping.

• cdp
• chrome-headless
• chrome-devtools
• chrome-devtools-protocol

🔥 The fastest and powerful Python library for Instagram Private API 2026 with HikerAPI SaaS

• instagram-api
• instagram
• instagram-client
• instagram-downloader

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

• web-scraping
• text-extraction
• nlp
• html2text

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

• crawler
• javascript
• spider
• extract-data

TranslatedA WeChat official account crawler interface based on Sogou WeChat search

• wechat
• sogou
• python
• crawler

The All in One Framework to Build Undefeatable Scrapers

• anti-bot
• anti-detection
• cloudflare-bypass
• cloudflare-scrape

TranslatedHands-on 🐍 web crawlers for many kinds of websites and e-commerce data 🕷. Includes 🕸: Taobao products, WeChat official accounts, Dianping, Qichacha, job recruitment sites, Xianyu, Alibaba tasks, Cnblogs, Weibo, Baidu Tieba, Douban movies, Baotu (ibaotu.com), Quanjing (quanjing.com), Douban music, a provincial drug administration, Sohu News, machine learning text collection, FOFA asset collection, Autohome, National Bureau of Statistics, Baidu keyword indexed-page counts, spider pool directories, Toutiao (Today's Headlines), Douban movie reviews, Ctrip, Xiaomi App Store, Anjuke, Tujia homestays ❤️❤️❤️. WeChat crawler showcase project:

• python3
• crawler
• baidu-tieba
• taobao-spider

Redis-based components for Scrapy.

• scrapy
• crawler
• distributed
• redis

Distributed crawler powered by Headless Chrome

• headless-chrome
• puppeteer
• jquery
• crawler

The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.

• mcp
• agent
• ai
• ai-agent

:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis

• high-availability
• scrapy
• ipproxy
• distributed

Dark Web OSINT Tool

• tor
• tor-network
• deepweb
• dark-web

Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application

• bugbounty
• crawling
• hacking
• osint

:zap: A distributed crawler for weibo, building with celery and requests.

• weibospider
• data-analysis
• python3
• distributed-crawler

TranslatedSina Weibo crawler: uses Python to crawl Sina Weibo data and download Weibo images and Weibo videos

• weibo
• crawler
• weibo-spider

Open Source Web Crawler for Java

Download comics novels 小说漫画下载工具 小説漫画のダウンローダ 小說漫畫下載:腾讯漫画 大角虫漫画 有妖气 咪咕 SF漫画 哦漫画 看漫画 漫画柜 汗汗酷漫 動漫伊甸園 快看漫画 微博动漫 733动漫网 大古漫画网 漫画DB 無限動漫 動漫狂 卡推漫画 动漫之家 动漫屋 古风漫画网 36漫画网 亲亲漫画网 乙女漫画 webtoons 咚漫 ニコニコ静画 ComicWalker ヤングエースUP モアイ pixivコミック サイコミ;アルファポリス カクヨム ハーメルン 小説家になろう 起点中文网 八一中文网 顶点小说 落霞小说网 努努书坊 笔趣阁→epub.

• comic-downloader
• novel-downloader
• cejs
• downloader

A list of AI agents and robots to block.

• ai
• crawlers
• crawling
• privacy

Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS :performing_arts:

• proxy
• anonymity
• privacy
• socks