Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
- agent-infrastructure
- ai-agent
- ai-search
- automation
Repository Index
47,133 repositories indexed
1–24 of 49 · Most stars
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
- crawler
- crawling
- crawling-python
- playwright
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
AI agent skill that researches any topic across Reddit, X, YouTube, HN, Polymarket, and the web - then synthesizes a grounded summary
- ai-prompts
- claude
- claude-code
Scrapy, a fast high-level web crawling & scraping framework for Python.
- python
- scraping
- crawling
- framework
Your lightweight open-source personal agent.
🕵️♂️ Collect a dossier on a person by username from 6K websites
- osint
- social-network
- identification
- socmint
Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring—all for free or enjoy our SaaS plan!
- website-monitor
- website-monitoring
- change-detection
- monitoring
Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 50+ tests passed.
- anti-detect
- bot-detection
- browser-automation
- chromium
Python scraper based on AI
- scraping
- scraping-python
- llm
- web-crawler
An autonomous agent that conducts deep research on any data using any LLM providers
- ai
- python
- agent
- automation
🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
- wget
- browser-bookmarks
- pinboard
Zulip server and web application. Open-source team chat that helps teams stay productive and focused.
- zulip
- python
- python3
- chat
Python ProxyPool for web spider
🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API，支持 AI agent 通过 MCP 调用与离线语音转文字，Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
🦉 OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automation
- agent
- artificial-intelligence
- multi-agent-systems
- task-automation
Turns Data and AI algorithms into production-ready web applications in no time.
Let Claude Code, Codex or Cursor use your real Chrome, logged in as you. One prompt to set up. Open source, MIT.
Air-gapped, privacy-focused, open-source NotebookLM alternative. AI agent that researches, transforms, and edits your documents. Join our Discord: https://discord.gg/ejRNvftDp9
🖥️ Run AI Agent in your browser.
- ai-agent
- browser-automation
- browser-use-box
- cloud-browser
AWX provides a web-based user interface, REST API, and task engine built on top of Ansible. It is one of the upstream projects for Red Hat Ansible Automation Platform.
Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection
Python version of the Playwright testing and automation library.
- playwright
- chromium
- firefox
- webkit
📊 APIs for Web Scraping, Browser Automation, and E2E Testing. CDP Mode & Stealthy Playwright Mode bypass bot-detection/CAPTCHAs.
- python
- selenium-python
- e2e-testing
- seleniumbase