TranslatedHands-on 🐍 web crawlers for many kinds of websites and e-commerce data 🕷. Includes 🕸: Taobao products, WeChat official accounts, Dianping, Qichacha, job recruitment sites, Xianyu, Alibaba tasks, Cnblogs, Weibo, Baidu Tieba, Douban movies, Baotu (ibaotu.com), Quanjing (quanjing.com), Douban music, a provincial drug administration, Sohu News, machine learning text collection, FOFA asset collection, Autohome, National Bureau of Statistics, Baidu keyword indexed-page counts, spider pool directories, Toutiao (Today's Headlines), Douban movie reviews, Ctrip, Xiaomi App Store, Anjuke, Tujia homestays ❤️❤️❤️. WeChat crawler showcase project:
- python3
- crawler
- baidu-tieba
- taobao-spider
Redis-based components for Scrapy.
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
Dark Web OSINT Tool
- tor
- tor-network
- deepweb
- dark-web
TranslatedMulti-platform content monitoring, collection, and reposting: one web panel to manage Douyin / Xiaohongshu / Kuaishou / WeChat Channels / TikTok
Find web directories without bruteforce
Async Python 3.6+ web scraping micro-framework based on asyncio
- asyncio
- aiohttp
- asyncio-spider
- crawler
The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.
- csrf
- crawler
- csrf-attacks
- token-generation
Generate smart and powerful wordlists
SEO foundation checks as an Agent Skill: give it a URL, get a crawler's-eye pass/fail report with evidence. 26 structural checks, zero LLM, deterministic.
Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities using realistic, randomly generated decoy data and AI-generated HTML templates.
Translated✈️ Self-hosted multi-channel airfare price monitoring: flight detail collection → DingTalk rich-media push → four-layer strong alerts when the target price is met. Domestic routes and international routes (destinations on five continents) share the same collect-judge-push pipeline; data stays on your own machine, all configuration is done on a web page, open source and free.
WarcDB: Web crawl data as SQLite databases.
TranslatedAll-in-one CLI & Skill to download anything: videos, docs, cloud drives, music, papers & books. An all-in-one download and archiving tool for AI Agents and terminals.
Ragpicker is a Plugin based malware crawler with pre-analysis and reporting functionalities. Use this tool if you are testing antivirus products, collecting malware for another analyzer/zoo.
A Python DNS crawler to find identical domain names under different TLDs.
German small and large versions of GPT2.