Repository Index

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

• ai
• crawler
• markdown
• scraper

Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.

• agent-infrastructure
• ai-agent
• ai-search
• automation

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

• ai
• ai-agents
• crawler
• data-extraction

Python scraper based on AI

• scraping
• scraping-python
• llm
• web-crawler

🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

• pocket
• wget
• browser-bookmarks
• pinboard

TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.

• tiktok
• douyin
• scraper
• crawler

Turn any website into a structured API. Extract, automate, search and monitor the web.

• automation
• no-code
• scraper
• web-scraper

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

• crawler
• javascript
• spider
• extract-data

An archiving tool with an IM-style interface that prioritizes privacy and accessibility, integrated with various archival services including Internet Archive, archive.today, Ghostarchive, IPFS, Telegraph, and file systems.

• telegram
• archive
• wayback-machine
• save-the-internet

❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Europe on platforms like ImmoScout24, Immowelt, eBay Kleinanzeigen and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)

• fredy
• immowelt
• appartment
• appartments

Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.

• java
• elasticsearch
• full-text-search
• lucene

Web fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.

• agpl
• ai-agents
• anti-bot
• crawling

WarcDB: Web crawl data as SQLite databases.

• crawling
• sqlite
• warc
• cli

Zotero extension that combats link rot by archiving webpages and journal articles.

DocSearch - Scraper

• algolia
• docsearch
• documentation
• scraper

Extract web archive data using Wayback Machine and Common Crawl

• golang
• crawler
• commoncrawl
• wayback-machine

A robust web archive analytics toolkit

• python
• web
• warc
• bigdata

An app to search startup jobs scraped from websites written in Elixir, Phoenix, React and styled-components.

• elixir
• phoenix
• otp
• react

Search the web and primary sources, read full pages, and check every citation. MCP server for Claude, Cursor, Codex and more. Runs on your machine.

• ai
• claude
• content-extraction
• go

Webpage summary extractor using Facebook Open Graph and arc90's readability

• readability
• scraper
• opengraph

Browser extension to simulate browsing behaviour in search engines.

• automation
• audit
• browser-extension
• firefox