Repository Index

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

• ai
• crawler
• markdown
• scraper

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

• ai
• ai-agents
• crawler
• data-extraction

Python scraper based on AI

• scraping
• scraping-python
• llm
• web-crawler

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

• web-scraping
• web-crawling
• npm
• headless-chrome

Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust

• browser
• rust
• servo
• browser-automation

TranslatedSome interesting examples of python crawlers that are friendly to beginners. Mainly crawls websites such as 淘宝 (Taobao, "searching for treasure"), 天猫 (Tmall, "sky cat"), 微信 (WeChat, "micro message"), 微信读书 (WeChat Reading), 豆瓣 (Douban, "bean petal"), and QQ.

• crawler
• spider
• taobao
• tmall

TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.

• webcrawler
• scrapy
• crawlab
• spiders-management

TranslatedA new-generation web crawler platform that defines crawler workflows graphically, so you can build a crawler without writing code.

• spider
• crawler
• jsoup
• xpath

Dark Web OSINT Tool

• tor
• tor-network
• deepweb
• dark-web

A scalable, mature and versatile web crawler based on Apache Storm

• web-crawler
• apache-storm
• distributed
• java

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

• spider
• ruby
• spider-links
• crawler

The unix-way web crawler

• cli
• crawler
• golang-application
• unix-way

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

• golang
• web-crawler
• crawling
• scraping

Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.

• ai-agents
• browser-automation
• data-extraction
• firecrawl

Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.

• crawling
• csharp
• elixir
• ffi