Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Repository Index
47,133 repositories indexed
1–8 of 8 · Most stars
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
- crawler
- crawling
- crawling-python
- playwright
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
Python scraper based on AI
- scraping
- scraping-python
- llm
- web-crawler
Turn any website into a structured API. Extract, automate, search and monitor the web.
- automation
- no-code
- scraper
- web-scraper
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
- python
- news
- crawler
- crawling
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
- web-scraping
- text-extraction
- nlp
- html2text
Rust web scraper & crawler for AI agents: URLs, PDFs and Office files → clean Markdown/JSON. Firecrawl- and Jina Reader-compatible API, MCP server, Python/Node SDKs. Self-hosted single binary.
- ai-agents
- browser-automation
- data-extraction
- firecrawl