Web Crawler/Spider for NodeJS + server-side jQuery ;-)
Repository Index
55,640 visible repositories in the index
49–72 of 133 · Most stars
Shattered Pixel Dungeon is an open-source traditional roguelike dungeon crawler with randomized levels and enemies, and hundreds of items to collect and use. It's based on the source code of Pixel Dungeon, by Watabou.
TranslatedA WeChat official account crawler interface based on Sogou WeChat search
An open source deep research clone. AI Agent that reasons large amounts of web data extracted with Firecrawl
Declarative data automation language and Go runtime for structured extraction workflows.
- golang
- query-language
- dsl
- go
The All in One Framework to Build Undefeatable Scrapers
- anti-bot
- anti-detection
- cloudflare-bypass
- cloudflare-scrape
TranslatedHands-on 🐍 web crawlers for many kinds of websites and e-commerce data 🕷. Includes 🕸: Taobao products, WeChat official accounts, Dianping, Qichacha, job recruitment sites, Xianyu, Alibaba tasks, Cnblogs, Weibo, Baidu Tieba, Douban movies, Baotu (ibaotu.com), Quanjing (quanjing.com), Douban music, a provincial drug administration, Sohu News, machine learning text collection, FOFA asset collection, Autohome, National Bureau of Statistics, Baidu keyword indexed-page counts, spider pool directories, Toutiao (Today's Headlines), Douban movie reviews, Ctrip, Xiaomi App Store, Anjuke, Tujia homestays ❤️❤️❤️. WeChat crawler showcase project:
- python3
- crawler
- baidu-tieba
- taobao-spider
Redis-based components for Scrapy.
- scrapy
- crawler
- distributed
- redis
Distributed crawler powered by Headless Chrome
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
- high-availability
- scrapy
- ipproxy
- distributed
Dark Web OSINT Tool
- tor
- tor-network
- deepweb
- dark-web
Open Source Web Crawler for Java
A self-hosted BitTorrent indexer, DHT crawler, content classifier and torrent search engine with web UI, GraphQL API and Servarr stack integration.
- bittorrent
- dht
- prowlarr
- radarr
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
- crawler
- dotnetcore
- cross-platform
- csharp
Headless Chrome .NET API
- puppeteer
- chrome
- chromium
- automation
TranslatedMulti-platform content monitoring, collection, and reposting: one web panel to manage Douyin / Xiaohongshu / Kuaishou / WeChat Channels / TikTok
- automation
- content-monitoring
- crawler
- douyin
Dungeon Crawl: Stone Soup official repository
A browser testing and web crawling library for PHP and Symfony
A curated list of awesome puppeteer resources.
- puppeteer
- headless-chrome
- awesome
- awesome-list
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
Find web directories without bruteforce
- pentesting
- security
- security-tools
- websec
Free database schema discovery and comprehension tool
- schemacrawler
- jdbc
- database-diagrams
- schema
Async Python 3.6+ web scraping micro-framework based on asyncio
- asyncio
- aiohttp
- asyncio-spider
- crawler