🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Repository Index
47,133 repositories indexed
1–12 of 12 · Most stars
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
Scrapy, a fast high-level web crawling & scraping framework for Python.
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
Incredibly fast crawler designed for OSINT.
Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机
Dark Web OSINT Tool
Find web directories without bruteforce
Async Python 3.6+ web scraping micro-framework based on asyncio
Generate smart and powerful wordlists
✈️ 自托管的多渠道机票价格监控：航班明细采集 → 钉钉图文推送 → 达标四层强提醒。国内航线与国际航线（五洲目的地）同一套采集-判定-推送链路；数据留在本机，配置全在网页，开源免费。
A Python DNS crawler to find identical domain names under different TLDs.