TranslatedA WeChat official account crawler interface based on Sogou WeChat search
Repository Index
72,362 visible repositories in the index
25–48 of 76 · Most stars
TranslatedHands-on 🐍 web crawlers for many kinds of websites and e-commerce data 🕷. Includes 🕸: Taobao products, WeChat official accounts, Dianping, Qichacha, job recruitment sites, Xianyu, Alibaba tasks, Cnblogs, Weibo, Baidu Tieba, Douban movies, Baotu (ibaotu.com), Quanjing (quanjing.com), Douban music, a provincial drug administration, Sohu News, machine learning text collection, FOFA asset collection, Autohome, National Bureau of Statistics, Baidu keyword indexed-page counts, spider pool directories, Toutiao (Today's Headlines), Douban movie reviews, Ctrip, Xiaomi App Store, Anjuke, Tujia homestays ❤️❤️❤️. WeChat crawler showcase project:
- python3
- crawler
- baidu-tieba
- taobao-spider
Redis-based components for Scrapy.
- scrapy
- crawler
- distributed
- redis
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
- high-availability
- scrapy
- ipproxy
- distributed
Dark Web OSINT Tool
- tor
- tor-network
- deepweb
- dark-web
TranslatedWeb crawler basics, intermediate web crawling, advanced web crawlers
:zap: A distributed crawler for weibo, building with celery and requests.
- weibospider
- data-analysis
- python3
- distributed-crawler
TranslatedSina Weibo crawler: uses Python to crawl Sina Weibo data and download Weibo images and Weibo videos
- crawler
- weibo-spider
TranslatedPython web crawler tutorial that takes you from zero to one, covering JS reverse engineering, Selenium, Tesseract OCR recognition, the use of MongoDB, and the Scrapy framework
The Ultimate Information Gathering Toolkit
- cms-detection
- directory-finder
- dns-lookup
- information-gathering
A list of AI agents and robots to block.
Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS :performing_arts:
Intelligent proxy pool for Humans™ to extract content from the internet and build your own Large Language Models in this new AI era
- crawler
- python3
- scylla
- proxy-pool
新闻网页正文通用抽取器 Beta 版.
- python3
- webcrawler
- webspider
🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度
Every web site provides APIs.
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
- scrapy
- distributed
- webspider
- scrapyd
Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️
- scrapy
- python
- crawl
- crawler
Videodl: A lightweight video downloader written in pure python. (轻量级视频下载器，优先高清无水印，支持抖音，快手，小红书，B站，TikTok，YouTube，FIFA+，优酷，腾讯，爱奇艺，1905电影网，乐视，芒果，咪咕，PPTV，搜狐，Facebook，Twitter，新浪微博，今日头条，网易公开课，全民K歌，CCTV央视频，酷狗音乐MV，新片场，知乎，百度贴吧，TED等海量流媒体平台)
- video-downloader
- cntv
- douyin
- crawler
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现
TranslatedMulti-platform content monitoring, collection, and reposting: one web panel to manage Douyin / Xiaohongshu / Kuaishou / WeChat Channels / TikTok
- automation
- content-monitoring
- crawler
- douyin
All In One Web Recon
- python3
- pentesting
- pentest-tool
- pentesting-tools
Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.
- scraper
- twitter-api
- twitter-bot
Learn step-by-step how to scrape Google Trends data and make a result comparison using Python and Oxylabs SERP API. Extract keywords, their popularity, breakdown by region, related queries, and more.
- google-trend
- google-trend-api
- google-trend-strategy