Repository Index

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

• ai
• crawler
• markdown
• scraper

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

• ai
• ai-agents
• crawler
• data-extraction

Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust

• browser
• rust
• servo
• browser-automation

TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.

• tiktok
• douyin
• scraper
• crawler

Turn any website into a structured API. Extract, automate, search and monitor the web.

• automation
• no-code
• scraper
• web-scraper

Web fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.

• agpl
• ai-agents
• anti-bot
• crawling

Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.

• ai-agents
• browser-automation
• data-extraction
• firecrawl