TranslatedA visual no-code/code-free web crawler/spider. 易采集 (EasySpider, "easy collection"): a visual browser automation testing/data collection/web crawler software that lets you design and execute crawler tasks graphically without code. Alias: ServiceWrapper, an intelligent service wrapping system for Web applications.
Repository Index
62,272 visible repositories in the index
1–24 of 51 · Most stars
Elegant Scraper and Crawler Framework for Golang
Python ProxyPool for web spider
A next-generation crawling and spidering framework.
- crawler
- web-spider
- gocrawler
- spider-framework
Incredibly fast crawler designed for OSINT.
TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.
- webcrawler
- scrapy
- crawlab
- spiders-management
TranslatedA new-generation web crawler platform that defines crawler workflows graphically, so you can build a crawler without writing code.
TranslatedAV movie management system, avmoo, javbus, javlibrary crawler, online AV movie library, AV magnet link database, Japanese Adult Video Library, Adult Video Magnet Links - Japanese Adult Video Database
- javbus
- avmoo
- javlibrary
- spider
TranslatedSina Weibo crawler, uses Python to scrape Sina Weibo data
- weibospider
- python
- python3
TranslatedXiaohongshu (Little Red Book) crawler data collection, Xiaohongshu reverse engineering, private messages, live streaming, full-domain operations solution for Xiaohongshu
TranslatedINFO-SPIDER is a crawler toolbox 🧰 that brings many data sources together in one place, designed to help users get their own data back safely and quickly. The tool's code is open source and the process is transparent. Supported data sources include GitHub, QQ Mail, NetEase Mail, Alibaba Mail, Sina Mail, Hotmail, Outlook Mail, JD.com, Taobao, Alipay, China Mobile, China Unicom, China Telecom, Zhihu, Bilibili, NetEase Cloud Music, QQ friends, QQ groups, generating a Moments (WeChat friend circle) photo album, browser browsing history, 12306, cnblogs (博客园), CSDN blog, OSChina blog, Jianshu.
Pholcus is a distributed high-concurrency crawler software written in pure golang
- spider
- crowler
TranslatedMore and more websites have anti-crawler features: some hide key data in images, and some use inhuman CAPTCHAs. This is a repository of anti-anti-crawler code, built to improve skills by contending with websites of different characteristics (with no malicious intent). (Submissions of websites that are hard to scrape are welcome.) (The project is paused due to work reasons.)
- python
- spider
- geek
TranslatedProvince, city, district/county and township level (three-level or four-level) city data, with pinyin annotations, coordinates, and administrative region boundary ranges; latest collection on April 3, 2026; provides csv format files, supports online conversion into multi-level linked js code and general json format, provides software to convert into shp, geojson, sql and import into databases; includes js collection source code that runs in the browser, combining administrative division data from the Ministry of Civil Affairs of the People's Republic of China, China National Geographical Names Information Library, the Bureau of Statistics, Amap (Gaode Map), and Tencent Maps
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
TranslatedSync articles to multiple content platforms with one click. Supports major platforms including Toutiao (今日头条), WordPress, Zhihu (知乎), Jianshu (简书), Juejin (掘金), CSDN, and typecho. Publish once, sync to multiple platforms. Free up personal productivity.
- chrome-extension
- chrome
- markdown
- multiplatform
An Efficient ProxyPool with Getter, Tester and Server
TranslatedHands-on 🐍 web crawlers for many kinds of websites and e-commerce data 🕷. Includes 🕸: Taobao products, WeChat official accounts, Dianping, Qichacha, job recruitment sites, Xianyu, Alibaba tasks, Cnblogs, Weibo, Baidu Tieba, Douban movies, Baotu (ibaotu.com), Quanjing (quanjing.com), Douban music, a provincial drug administration, Sohu News, machine learning text collection, FOFA asset collection, Autohome, National Bureau of Statistics, Baidu keyword indexed-page counts, spider pool directories, Toutiao (Today's Headlines), Douban movie reviews, Ctrip, Xiaomi App Store, Anjuke, Tujia homestays ❤️❤️❤️. WeChat crawler showcase project:
- python3
- crawler
- baidu-tieba
- taobao-spider
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
- high-availability
- scrapy
- ipproxy
- distributed
Dark Web OSINT Tool
- tor
- tor-network
- deepweb
- dark-web
:zap: A distributed crawler for weibo, building with celery and requests.
- weibospider
- data-analysis
- python3
- distributed-crawler
TranslatedToday's Hot List (今日热榜), an aggregator website that gets the hot headlines from major popular websites. Written in Go, it uses multiple goroutines to fetch information asynchronously and quickly. Preview: https://mo.fish
- hot
- golang
- spider
- today-s-hot-list
TranslatedSina Weibo crawler: uses Python to crawl Sina Weibo data and download Weibo images and Weibo videos
- crawler
- weibo-spider
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
- crawler
- dotnetcore
- cross-platform
- csharp