Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Repository Index
56,638 visible repositories in the index
1–24 of 57 · Most stars
Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
- agent-infrastructure
- ai-agent
- ai-search
- automation
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring— MCP included! all for free or enjoy our SaaS plan!
- website-monitor
- website-monitoring
- change-detection
- monitoring
Python scraper based on AI
- scraping
- scraping-python
- llm
- web-crawler
The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more.
Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
- browser
- rust
- servo
- browser-automation
TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
Turn any website into a structured API. Extract, automate, search and monitor the web.
- automation
- no-code
- scraper
- web-scraper
API Support for your favorite torrent trackers
⬛️ GUI/CLI tool and library for saving complete web pages as a single HTML file
- save-the-internet
- procrastination
- its-mine
- no-more-404
TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.
- webcrawler
- scrapy
- crawlab
- spiders-management
Get the main content of any page as Markdown.
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.
- apify
- automation
- beautifulsoup
- crawler
Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.
- python
- machine-learning
- natural-language-processing
- web-mining
Python Script to download hundreds of images from 'Google Images'. It is a ready-to-run code!
- python
- python-script
- image-download
- google-images
scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place
- golang
- google-maps-scraping
- web-scraper
- web-scraping
Turn any webpage into structured data using LLMs
The Unofficial TikTok API Wrapper In Python
- tiktok-api
- python
- api
- trending
💡 Download the complete source code of any website (including all assets). [ Javascripts, Stylesheets, Images ] using Node.js
- offline-web-pages
- downloader
- scraper
- assets
📜 Extract meaningful content from the chaos of a web page
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
Translated✨WeChat Official Account assistant that fits reading habits, WeChat Official Account to MarkDown, WeChat Official Account to PDF, scheduled updates of subscribed Official Account articles, generate RSS feeds for WeChat Official Accounts, export WeChat Official Account subscription feeds, supports Webhook/API/AI Agent integration for WeChat Official Accounts, subscribe to WeChat Official Accounts, WeChat Official Account assistant, WeChat Official Account reading, WeChat Official Account interface, WeChat Official Account crawler, WeChat Official Account monitoring, tag-based subscription to WeChat Official Accounts, WeChat Official Account source, WeChat Official Account book reading, WeChat Official Account articles, WeChat Official Account framework, WeChat Official Account management, WeChat Official Account source, WeChat Official Account platform, WeChat Official Account code, WeChat Official Account system, WeChat Official Account source code
- rss-feed
- rss-hub
- weichat
- weixin
Extract any website's complete design system with one command. DTCG tokens, semantic+primitive+composite, MCP server for Claude Code/Cursor/Windsurf, multi-platform emitters (iOS SwiftUI, Android Compose, Flutter, WordPress), Tailwind v4, Figma variables, shadcn/ui, CSS health audit, WCAG remediation, Chrome extension. MIT, Playwright, Node 20+.