🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Repository Index
47,133 repositories indexed
1–23 of 23 · Most stars
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
Scrapy, a fast high-level web crawling & scraping framework for Python.
TuShare is a utility for crawling historical data of China stocks
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )
Draw pretty maps from OpenStreetMap data! Built with osmnx +matplotlib + shapely
Incredibly fast crawler designed for OSINT.
Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机
将 SRT 字幕做成暖米黄纸张底的流式笔迹白板手绘动画 skill：mask 分区遮罩编排 + stream 连续笔迹（ink→color）。
Dark Web OSINT Tool
A GraphQL library for Python that leverages type annotations 🍓
Find web directories without bruteforce
Async Python 3.6+ web scraping micro-framework based on asyncio
Generate smart and powerful wordlists
Python interface to Graphviz graph drawing package
✈️ 自托管的多渠道机票价格监控：航班明细采集 → 钉钉图文推送 → 达标四层强提醒。国内航线与国际航线（五洲目的地）同一套采集-判定-推送链路；数据留在本机，配置全在网页，开源免费。
An extension for Inkscape that makes it easier to draw optical diagrams.
Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.
Fast, memory-safe camera RAW decoder in Rust: ~40 formats, SIMD, GPU, WebAssembly, ICC colour management. CLI, C, C++, Python and JavaScript bindings.
Open framework for calculating spatial urban sprawl indices and performing disaggregated population estimates using open data
Making graphs for yEd
A Python DNS crawler to find identical domain names under different TLDs.