Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
- agent-infrastructure
- ai-agent
- ai-search
- automation
107,098 visible repositories in the index
1–24 of 60 · Most stars
Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
🕵️♂️ Collect a dossier on a person by username from 6K websites
Python scraper based on AI
🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
Turn any website into a structured API. Extract, automate, search and monitor the web.
⬛️ GUI/CLI tool and library for saving complete web pages as a single HTML file
Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.
Turn any webpage into structured data using LLMs
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
Twitter API Scraper | Without an API key | Twitter Internal API | Free | Twitter scraper | Twitter Bot
Jobs scraper library for LinkedIn, Indeed, Glassdoor, ZipRecruiter, Bayt, Naukri, BDJobs, HelloWork & SEEK
Automatically crawls proxy nodes on the public internet, de-duplicates and tests for usability and then provides a list of nodes
Intelligent proxy pool for Humans™ to extract content from the internet and build your own Large Language Models in this new AI era
Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up
Abstraction layer over YouTube's internal API
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
Turn any website into an API your agent can call. Learns from the site's own frontend, heals itself when the site changes.
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!
AutoCLI is a Blazing fast, memory-safe command-line tool — Fetch information from any website with a single command. Covers Twitter/X, Reddit, YouTube, HackerNews, Bilibili, Zhihu, Xiaohongshu, and 55+ sites, with support for controlling Electron desktop apps, integrating local CLI tools (gh, docker, kubectl), now powered by AutoCLI.ai .
Point it at any web page and it finds the video, transcodes it, and streams it to your TV in real time. It even burns in subtitles.
📚 Turn any web page into a clean view