🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Repository Index
47,133 repositories indexed
1–24 of 51 · Most stars
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
小红书笔记 | 评论爬虫、抖音视频 | 评论爬虫、快手视频 | 评论爬虫、B 站视频 ｜ 评论爬虫、微博帖子 ｜ 评论爬虫、百度贴吧帖子 ｜ 百度贴吧评论回复爬虫 | 知乎问答文章｜评论爬虫
Scrapy, a fast high-level web crawling & scraping framework for Python.
Python scraper based on AI
Python ProxyPool for web spider
🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API，支持 AI agent 通过 MCP 调用与离线语音转文字，Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
TuShare is a utility for crawling historical data of China stocks
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
Draw pretty maps from OpenStreetMap data! Built with osmnx +matplotlib + shapely
Incredibly fast crawler designed for OSINT.
Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
将 SRT 字幕做成暖米黄纸张底的流式笔迹白板手绘动画 skill：mask 分区遮罩编排 + stream 连续笔迹（ink→color）。
Dark Web OSINT Tool
A GraphQL library for Python that leverages type annotations 🍓
Find web directories without bruteforce
Async Python 3.6+ web scraping micro-framework based on asyncio
Convert your KiCAD board into a nicely looking 2D drawing suitable for pinout diagrams
The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.
Generate smart and powerful wordlists
SEO foundation checks as an Agent Skill: give it a URL, get a crawler's-eye pass/fail report with evidence. 26 structural checks, zero LLM, deterministic.
Convert images to vectorized line drawings for plotters.
Python interface to Graphviz graph drawing package