TranslatedA visual no-code/code-free web crawler/spider. 易采集 (EasySpider, "easy collection"): a visual browser automation testing/data collection/web crawler software that lets you design and execute crawler tasks graphically without code. Alias: ServiceWrapper, an intelligent service wrapping system for Web applications.
Repository Index
132,241 visible repositories in the index
1–24 of 286 · Most stars
Elegant Scraper and Crawler Framework for Golang
SpiderFoot automates OSINT for threat intelligence and mapping your attack surface.
- footprinting
- osint
- threatintel
- python
Python ProxyPool for web spider
TranslatedA Python web scraping tutorial series, learning Python web scraping from 0 to 1, including browser packet capture and mobile APP packet capture with tools such as fiddler and mitmproxy, the use of the various modules involved in web scraping, such as requests, beautifulSoup, selenium, appium, scrapy, etc., as well as IP proxies, CAPTCHA recognition, using Mysql and MongoDB databases with Python, multi-threaded and multi-process scraping, reverse engineering and cracking of CSS scraping encryption, JS scraping reverse engineering, distributed scraping, practical scraping project examples, etc.
- python3
- python-script
- python-spider
Translated:rainbow:Python3 web crawler practice: Taobao, JD.com, NetEase Cloud Music, Bilibili, 12306, Douyin, Biquge, comic and novel downloads, music and movie downloads, etc.
- python-spider
- python3
- python
- webspider
A next-generation crawling and spidering framework.
- crawler
- web-spider
- gocrawler
- spider-framework
Translated😮Python simulated login for some large websites, plus some simple web crawlers. I hope they are helpful to you❤️. If you like it, remember to give it a star🌟
- python
- spiders
- selenium
- twitter-bot
TranslatedSome interesting examples of python crawlers that are friendly to beginners. Mainly crawls websites such as 淘宝 (Taobao, "searching for treasure"), 天猫 (Tmall, "sky cat"), 微信 (WeChat, "micro message"), 微信读书 (WeChat Reading), 豆瓣 (Douban, "bean petal"), and QQ.
Incredibly fast crawler designed for OSINT.
TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.
- webcrawler
- scrapy
- crawlab
- spiders-management
TranslatedPython open-source project "The Road to Self-Taught Programming", with step-by-step tutorials for beginners: AI Lab, treasure videos, data structures, study guides, hands-on machine learning, hands-on deep learning, web crawlers, big-company interview experiences, programmer life, resource sharing.
- python3
- pytorch
- deeplearning
- deep-learning
TranslatedA new-generation web crawler platform that defines crawler workflows graphically, so you can build a crawler without writing code.
TranslatedLive stream recording software that supports looped monitoring and recording multiple streamers at once. It supports recording live streams from 40+ platforms, including Douyin, TikTok, Youtube, Kuaishou, Huya, Douyu, Bilibili, Xiaohongshu, pandatv, sooplive, flextv, popkontv, twitcasting, winktv, Baidu, Weibo, Kugou, 17Live, Twitch, Acfun, CHZZK, shopee, and more
- douyin-live
- douyin-api
- douyulive
- live-recorder
TranslatedAV movie management system, avmoo, javbus, javlibrary crawler, online AV movie library, AV magnet link database, Japanese Adult Video Library, Adult Video Magnet Links - Japanese Adult Video Database
- javbus
- avmoo
- javlibrary
- spider
TranslatedSina Weibo crawler, uses Python to scrape Sina Weibo data
- weibospider
- python
- python3
TranslatedXiaohongshu (Little Red Book) crawler data collection, Xiaohongshu reverse engineering, private messages, live streaming, full-domain operations solution for Xiaohongshu
TranslatedINFO-SPIDER is a crawler toolbox 🧰 that brings many data sources together in one place, designed to help users get their own data back safely and quickly. The tool's code is open source and the process is transparent. Supported data sources include GitHub, QQ Mail, NetEase Mail, Alibaba Mail, Sina Mail, Hotmail, Outlook Mail, JD.com, Taobao, Alipay, China Mobile, China Unicom, China Telecom, Zhihu, Bilibili, NetEase Cloud Music, QQ friends, QQ groups, generating a Moments (WeChat friend circle) photo album, browser browsing history, 12306, cnblogs (博客园), CSDN blog, OSChina blog, Jianshu.
List of libraries, tools and APIs for web scraping and data processing.
- web-scraping
- captcha-bypass
- captcha-recaptcha
- crawling
Pholcus is a distributed high-concurrency crawler software written in pure golang
- spider
- crowler
TranslatedThe essence edition of getting started with web crawlers in Python
A collection of awesome web crawler,spider in different languages
TranslatedMore and more websites have anti-crawler features: some hide key data in images, and some use inhuman CAPTCHAs. This is a repository of anti-anti-crawler code, built to improve skills by contending with websites of different characteristics (with no malicious intent). (Submissions of websites that are hard to scrape are welcome.) (The project is paused due to work reasons.)
- python
- spider
- geek
TranslatedProvince, city, district/county and township level (three-level or four-level) city data, with pinyin annotations, coordinates, and administrative region boundary ranges; latest collection on April 3, 2026; provides csv format files, supports online conversion into multi-level linked js code and general json format, provides software to convert into shp, geojson, sql and import into databases; includes js collection source code that runs in the browser, combining administrative division data from the Ministry of Civil Affairs of the People's Republic of China, China National Geographical Names Information Library, the Bureau of Statistics, Amap (Gaode Map), and Tencent Maps