47,133 repositories indexed
1–24 of 51 · Most stars
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
TranslatedXiaohongshu notes | comment crawler, Douyin videos | comment crawler, Kuaishou videos | comment crawler, Bilibili videos | comment crawler, Weibo posts | comment crawler, Baidu Tieba posts | Baidu Tieba comment and reply crawler | Zhihu Q&A and articles | comment crawler
Scrapy, a fast high-level web crawling & scraping framework for Python.
Python scraper based on AI
Python ProxyPool for web spider
TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
TuShare is a utility for crawling historical data of China stocks
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
Draw pretty maps from OpenStreetMap data! Built with osmnx +matplotlib + shapely
Incredibly fast crawler designed for OSINT.
TranslatedPython scripts. Simulated login to Zhihu, web crawlers, working with Excel, WeChat official accounts, remote power-on
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
TranslatedA skill that turns SRT subtitles into whiteboard hand-drawn animation with a warm beige paper background and flowing handwriting strokes: mask-based zone masking and arrangement + stream continuous strokes (ink→color).
Dark Web OSINT Tool
A GraphQL library for Python that leverages type annotations 🍓
Find web directories without bruteforce
Async Python 3.6+ web scraping micro-framework based on asyncio
Convert your KiCAD board into a nicely looking 2D drawing suitable for pinout diagrams
The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.
Generate smart and powerful wordlists
SEO foundation checks as an Agent Skill: give it a URL, get a crawler's-eye pass/fail report with evidence. 26 structural checks, zero LLM, deterministic.
Convert images to vectorized line drawings for plotters.
Python interface to Graphviz graph drawing package