Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Repository Index
47,133 repositories indexed
1–10 of 10 · Most stars
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
Python scraper based on AI
- scraping
- scraping-python
- llm
- web-crawler
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
- web-scraping
- text-extraction
- nlp
- html2text
A curated list of awesome system integration software and resources.
- esb
- bpm
- ipaas
- apimanagement
🐘 A PHP client for interacting with Gotenberg.
Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.
- crawling
- csharp
- elixir
- ffi
Rust web scraper & crawler for AI agents: URLs, PDFs and Office files → clean Markdown/JSON. Firecrawl- and Jina Reader-compatible API, MCP server, Python/Node SDKs. Self-hosted single binary.
- ai-agents
- browser-automation
- data-extraction
- firecrawl
An awesome, curated list of various software testing related blogs and similar resources.
Confluence Cloud to Markdown crawler with incremental updates and local link rewriting