Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Repository Index
73,362 visible repositories in the index
1–24 of 86 · Most stars
Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
- agent-infrastructure
- ai-agent
- ai-search
- automation
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
🕵️♂️ Collect a dossier on a person by username from 6K websites
- osint
- social-network
- identification
- socmint
Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring— MCP included! all for free or enjoy our SaaS plan!
- website-monitor
- website-monitoring
- change-detection
- monitoring
The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more.
Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
- browser
- rust
- servo
- browser-automation
TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
Turn any website into a structured API. Extract, automate, search and monitor the web.
- automation
- no-code
- scraper
- web-scraper
API Support for your favorite torrent trackers
Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection
Download pictures (or videos) along with their captions and other metadata from Instagram.
- instagram-client
- instagram-feed
- downloader
TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.
- webcrawler
- scrapy
- crawlab
- spiders-management
Get the main content of any page as Markdown.
Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.
- python
- machine-learning
- natural-language-processing
- web-mining
scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place
- golang
- google-maps-scraping
- web-scraper
- web-scraping
An Efficient ProxyPool with Getter, Tester and Server
Download an entire website from the Wayback Machine.
📜 Extract meaningful content from the chaos of a web page
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
Translated✨WeChat Official Account assistant that fits reading habits, WeChat Official Account to MarkDown, WeChat Official Account to PDF, scheduled updates of subscribed Official Account articles, generate RSS feeds for WeChat Official Accounts, export WeChat Official Account subscription feeds, supports Webhook/API/AI Agent integration for WeChat Official Accounts, subscribe to WeChat Official Accounts, WeChat Official Account assistant, WeChat Official Account reading, WeChat Official Account interface, WeChat Official Account crawler, WeChat Official Account monitoring, tag-based subscription to WeChat Official Accounts, WeChat Official Account source, WeChat Official Account book reading, WeChat Official Account articles, WeChat Official Account framework, WeChat Official Account management, WeChat Official Account source, WeChat Official Account platform, WeChat Official Account code, WeChat Official Account system, WeChat Official Account source code
- rss-feed
- rss-hub
- weichat
- weixin
Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application
SwiftSoup: Pure Swift HTML Parser, with best of DOM, CSS, and jquery (Supports Linux, iOS, Mac, tvOS, watchOS)
- swift
- swiftsoup
- parse
- html-document
Extract any website's complete design system with one command. DTCG tokens, semantic+primitive+composite, MCP server for Claude Code/Cursor/Windsurf, multi-platform emitters (iOS SwiftUI, Android Compose, Flutter, WordPress), Tailwind v4, Figma variables, shadcn/ui, CSS health audit, WCAG remediation, Chrome extension. MIT, Playwright, Node 20+.