Repository Index

Python ProxyPool for web spider

• crawler
• proxy
• spider
• redis

Incredibly fast crawler designed for OSINT.

• crawler
• spider
• python
• osint

TranslatedSina Weibo crawler, uses Python to scrape Sina Weibo data

• weibo
• weibospider
• python
• python3

TranslatedXiaohongshu (Little Red Book) crawler data collection, Xiaohongshu reverse engineering, private messages, live streaming, full-domain operations solution for Xiaohongshu

TranslatedINFO-SPIDER is a crawler toolbox 🧰 that brings many data sources together in one place, designed to help users get their own data back safely and quickly. The tool's code is open source and the process is transparent. Supported data sources include GitHub, QQ Mail, NetEase Mail, Alibaba Mail, Sina Mail, Hotmail, Outlook Mail, JD.com, Taobao, Alipay, China Mobile, China Unicom, China Telecom, Zhihu, Bilibili, NetEase Cloud Music, QQ friends, QQ groups, generating a Moments (WeChat friend circle) photo album, browser browsing history, 12306, cnblogs (博客园), CSDN blog, OSChina blog, Jianshu.

• python3
• crawl
• spider
• selenium

TranslatedMore and more websites have anti-crawler features: some hide key data in images, and some use inhuman CAPTCHAs. This is a repository of anti-anti-crawler code, built to improve skills by contending with websites of different characteristics (with no malicious intent). (Submissions of websites that are hard to scrape are welcome.) (The project is paused due to work reasons.)

• python
• spider
• geek

An Efficient ProxyPool with Getter, Tester and Server

• proxypool
• redis
• http
• flask

TranslatedHands-on 🐍 web crawlers for many kinds of websites and e-commerce data 🕷. Includes 🕸: Taobao products, WeChat official accounts, Dianping, Qichacha, job recruitment sites, Xianyu, Alibaba tasks, Cnblogs, Weibo, Baidu Tieba, Douban movies, Baotu (ibaotu.com), Quanjing (quanjing.com), Douban music, a provincial drug administration, Sohu News, machine learning text collection, FOFA asset collection, Autohome, National Bureau of Statistics, Baidu keyword indexed-page counts, spider pool directories, Toutiao (Today's Headlines), Douban movie reviews, Ctrip, Xiaomi App Store, Anjuke, Tujia homestays ❤️❤️❤️. WeChat crawler showcase project:

• python3
• crawler
• baidu-tieba
• taobao-spider

:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis

• high-availability
• scrapy
• ipproxy
• distributed

Dark Web OSINT Tool

• tor
• tor-network
• deepweb
• dark-web

:zap: A distributed crawler for weibo, building with celery and requests.

• weibospider
• data-analysis
• python3
• distributed-crawler

TranslatedSina Weibo crawler: uses Python to crawl Sina Weibo data and download Weibo images and Weibo videos

• weibo
• crawler
• weibo-spider

持续维护的新浪微博采集工具🚀🚀🚀

• scrapy
• python
• weibo
• weibospider

新闻网页正文通用抽取器 Beta 版.

• python3
• webcrawler
• webspider

🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度

• scrapy
• feapder
• spider
• crawler

抖音逆向，抖音爬虫，抖音全部api、私信、直播间监听

Every web site provides APIs.

• html
• json
• api
• python

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

• scrapy
• distributed
• webspider
• scrapyd

微信公众号文章的爬虫

• wechat
• spider
• officialaccounts
• wechat-official-account

微信公众号爬虫

An environmental monitoring and regulation system

• raspberry-pi
• sensors
• regulation
• monitoring

The repository host gives no description.

Async Python 3.6+ web scraping micro-framework based on asyncio

• asyncio
• aiohttp
• asyncio-spider
• crawler