Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
Repository Index
47,133 repositories indexed
1–8 of 8 · Most stars
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
Peach: Local-first video metadata scraper and media manager.
:scissors: High performance, multi-threaded image scraper
Extract knowledge from all information sources using gpt and other language models. Index and make Q&A session with information sources.