🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Repository Index
47,133 repositories indexed
1–20 of 20 · Most stars
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
Scrapy, a fast high-level web crawling & scraping framework for Python.
TuShare is a utility for crawling historical data of China stocks
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
Draw pretty maps from OpenStreetMap data! Built with osmnx +matplotlib + shapely
Incredibly fast crawler designed for OSINT.
Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机
将 SRT 字幕做成暖米黄纸张底的流式笔迹白板手绘动画 skill：mask 分区遮罩编排 + stream 连续笔迹（ink→color）。
Dark Web OSINT Tool
A GraphQL library for Python that leverages type annotations 🍓
Find web directories without bruteforce
Async Python 3.6+ web scraping micro-framework based on asyncio
Generate smart and powerful wordlists
Python interface to Graphviz graph drawing package
✈️ 自托管的多渠道机票价格监控：航班明细采集 → 钉钉图文推送 → 达标四层强提醒。国内航线与国际航线（五洲目的地）同一套采集-判定-推送链路；数据留在本机，配置全在网页，开源免费。
An extension for Inkscape that makes it easier to draw optical diagrams.
Open framework for calculating spatial urban sprawl indices and performing disaggregated population estimates using open data
Making graphs for yEd
A Python DNS crawler to find identical domain names under different TLDs.