Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
- web-scraping
- text-extraction
- nlp
- html2text
Repository Index
72,362 visible repositories in the index
49–72 of 224 · Most stars
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
Shattered Pixel Dungeon is an open-source traditional roguelike dungeon crawler with randomized levels and enemies, and hundreds of items to collect and use. It's based on the source code of Pixel Dungeon, by Watabou.
n8n skillset for Claude Code to build flawless n8n workflows
TranslatedA WeChat official account crawler interface based on Sogou WeChat search
An open source deep research clone. AI Agent that reasons large amounts of web data extracted with Firecrawl
Declarative data automation language and Go runtime for structured extraction workflows.
- golang
- query-language
- dsl
- go
The All in One Framework to Build Undefeatable Scrapers
- anti-bot
- anti-detection
- cloudflare-bypass
- cloudflare-scrape
TranslatedHands-on 🐍 web crawlers for many kinds of websites and e-commerce data 🕷. Includes 🕸: Taobao products, WeChat official accounts, Dianping, Qichacha, job recruitment sites, Xianyu, Alibaba tasks, Cnblogs, Weibo, Baidu Tieba, Douban movies, Baotu (ibaotu.com), Quanjing (quanjing.com), Douban music, a provincial drug administration, Sohu News, machine learning text collection, FOFA asset collection, Autohome, National Bureau of Statistics, Baidu keyword indexed-page counts, spider pool directories, Toutiao (Today's Headlines), Douban movie reviews, Ctrip, Xiaomi App Store, Anjuke, Tujia homestays ❤️❤️❤️. WeChat crawler showcase project:
- python3
- crawler
- baidu-tieba
- taobao-spider
Redis-based components for Scrapy.
- scrapy
- crawler
- distributed
- redis
Distributed crawler powered by Headless Chrome
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
- high-availability
- scrapy
- ipproxy
- distributed
Exa MCP for web search and web crawling!
- code-search
- codesearch
- crawling
- mcp
Dark Web OSINT Tool
- tor
- tor-network
- deepweb
- dark-web
Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application
Fetch known URLs from AlienVault's Open Threat Exchange, the Wayback Machine, and Common Crawl.
- gau
- wayback-machine
- hacktoberfest
- security
Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️♂️ when scraping the web?
TranslatedSome interesting Python crawlers and data analysis projects
A Mighty CLI for AWS
TranslatedWeb crawler basics, intermediate web crawling, advanced web crawlers
:zap: A distributed crawler for weibo, building with celery and requests.
- weibospider
- data-analysis
- python3
- distributed-crawler
TranslatedCollection of China illegal cases about web crawler. This project compiles news, materials, and laws and regulations related to lawsuits and violations involving web crawler developers in mainland China. It aims to help web crawler industry practitioners working in mainland China understand China's relevant laws and avoid crossing the red lines of data compliance.
- china
- law
- crawler
TranslatedSina Weibo crawler: uses Python to crawl Sina Weibo data and download Weibo images and Weibo videos
- crawler
- weibo-spider