Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
- agent-infrastructure
- ai-agent
- ai-search
- automation
107,098 visible repositories in the index
1–24 of 171 · Most stars
Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
🕵️♂️ Collect a dossier on a person by username from 6K websites
Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring— MCP included! all for free or enjoy our SaaS plan!
🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more.
Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
Turn any website into a structured API. Extract, automate, search and monitor the web.
API Support for your favorite torrent trackers
⬛️ GUI/CLI tool and library for saving complete web pages as a single HTML file
Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
Download pictures (or videos) along with their captions and other metadata from Instagram.
Pythonic HTML Parsing for Humans™
Fast, reliable browser automation for web crawling, testing, scraping, and stealth. Bypasses CAPTCHAs with CDP Mode. Integrations include pytest, Playwright, MCP, and CLI.
jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety.
TranslatedAV movie management system, avmoo, javbus, javlibrary crawler, online AV movie library, AV magnet link database, Japanese Adult Video Library, Adult Video Magnet Links - Japanese Adult Video Database
Get the main content of any page as Markdown.
Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.
Python Script to download hundreds of images from 'Google Images'. It is a ready-to-run code!
This is a python API which allows you to get the transcript/subtitles for a given YouTube video. It also works for automatically generated subtitles and it does not require an API key nor a headless browser, like other selenium based solutions do!
The AI local media metadata crawler
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.