Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Repository Index
72,362 visible repositories in the index
1–12 of 12 · Most stars
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Convert Word, PowerPoint, Excel, OpenDocument, RTF, EPUB, CSV, and PDF to clean Markdown. Built in Rust, with Node.js and Python bindings.
Ingest, parse, and optimize any data format ➡️ from documents to multimedia ➡️ for enhanced compatibility with GenAI frameworks
- ingestion-api
- ocr
- omniparser
- parse-server
Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
- aitools
- crawl
- scrape
- serp
Leaked GPTs Prompts Bypass the 25 message limit or to try out GPTs without a Plus subscription.
- ai
- awesome
- awesome-list
- gpts
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
Web Crawling and RAG Capabilities for AI Agents and AI Coding Assistants
Transform Web Content into LLM-Ready Data
- crawl4ai
- crawler
- crawling-python
- html2markdown
Completely free, private, UI based Tech Documentation MCP server. Designed for coders and software developers in mind. Easily integrate into Cursor, Windsurf, Cline, Roo Code, Claude Desktop App
- crawl4ai
- documentation
- llm
- playwright
Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.
- ai-agents
- browser-automation
- data-extraction
- firecrawl