Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
- agent-infrastructure
- ai-agent
- ai-search
- automation
Repository Index
47,133 repositories indexed
1–15 of 15 · Most stars
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 50+ tests passed.
- anti-detect
- bot-detection
- browser-automation
- chromium
🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
- wget
- browser-bookmarks
- pinboard
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
- web-scraping
- text-extraction
- nlp
- html2text
Headless chrome/chromium automation library (unofficial port of puppeteer)
- automation
- puppeteer
- chromium
Playwright that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Python web scraping and browser automation.
- datadome
- perimeterx
- akamai
- antidetect-browser
:scissors: High performance, multi-threaded image scraper
A Python Package for the Google Chrome Dev Protocol [threading base]
- chrome-debugging-protocol
- python
- chrome
- headless
A Python feed reader library.
- rss
- rss-reader
- rss-aggregator
- rss-feed
Headless e-commerce framework for Django and Wagtail.
Previews for headless Wagtail setups
Credential gateway for AI agents. Log in once via Oauth2 or API Key. Every agent stays authenticated — headless, no SaaS, agents never see your credentials.
Selenium webdriver step library for use with the behave BDD testing framework
- selenium-webdriver
- webdriverio
- behave
- bdd
Python toolkit for discovering, monitoring, and controlling Tasmota devices via MQTT with CLI utilities and ready-to-use multi-arch Docker images
- automation
- cli
- control
- device-management