Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Repository Index
78,351 visible repositories in the index
1–15 of 15 · Most stars
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Air-gapped, privacy-focused, open-source NotebookLM alternative. AI agent that researches, transforms, and edits your documents. Join our Discord: https://discord.gg/ejRNvftDp9
Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection
Professional-grade, local-first browser extension that transforms any webpage into high-fidelity, LLM-friendly Markdown entirely within the browser's runtime. Zero-cloud, privacy-centric, and designed specifically for AI workflows.
- ai-workflows
- browser-extension
- chrome-extension-
- clipboard-api
Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
- aitools
- crawl
- scrape
- serp
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
Transform Web Content into LLM-Ready Data
- crawl4ai
- crawler
- crawling-python
- html2markdown
Specify a github or local repo, github pull request, arXiv or Sci-Hub paper, Youtube transcript or documentation URL on the web and scrape into a text file and clipboard for easier LLM ingestion
Turn any website into a compact CLI tailored for AI agents. Browse the web in hundreds of tokens, not tens of thousands.
Extract knowledge from all information sources using gpt and other language models. Index and make Q&A session with information sources.
Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.
- ai-agents
- browser-automation
- data-extraction
- firecrawl
Search and read the web for AI agents at 99% fewer tokens. Free CLI + MCP.
MindAct: Community-Driven AI Agent for Claude, Obsidian, and RAG Knowledge Graphs
- activator
- analytics
- artificial-inteligence
- bigquery