Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Repository Index
47,133 repositories indexed
1–24 of 44 · Most stars
Open-source Python library for AI browser agents, plus Browser Use Cloud: stealth browsers and a hosted agent API.
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
- crawler
- crawling
- crawling-python
- playwright
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. Join our Discord https://discord.gg/VuxzM8mtf
Scrapy, a fast high-level web crawling & scraping framework for Python.
- python
- scraping
- crawling
- framework
Clone any website with one command using AI coding agents
Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring—all for free or enjoy our SaaS plan!
- website-monitor
- website-monitoring
- change-detection
- monitoring
Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 50+ tests passed.
- anti-detect
- bot-detection
- browser-automation
- chromium
Python scraper based on AI
- scraping
- scraping-python
- llm
- web-crawler
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
- web-scraping
- web-crawling
- npm
- headless-chrome
The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more.
Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
- browser
- rust
- servo
- browser-automation
🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API，支持 AI agent 通过 MCP 调用与离线语音转文字，Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
Turn any website into a structured API. Extract, automate, search and monitor the web.
- automation
- no-code
- scraper
- web-scraper
Air-gapped, privacy-focused, open-source NotebookLM alternative. AI agent that researches, transforms, and edits your documents. Join our Discord: https://discord.gg/ejRNvftDp9
Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection
Open-source AI agent for web automation. Use your real, logged-in browser to automate daily tasks.
📊 APIs for Web Scraping, Browser Automation, and E2E Testing. CDP Mode bypasses bot-detection and handles CAPTCHAs. Adds stealth to Playwright. Includes an MCP server.
- python
- selenium-python
- e2e-testing
- seleniumbase
Stealth headless browser for AI agents — bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.
- ai-agent
- anti-bot
- antidetect-browser
- automation
jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety.
Automated CAPTCHA solver for your browser. Works with Selenium, Puppeteer, Playwright, and more.
- perimeterx
- nopecha
- ai-agents
- anti-captcha
Lighter web automation with Python
- selenium
- selenium-python
- python
- python3
A Chrome DevTools Protocol driver for web automation and scraping.
- cdp
- chrome-headless
- chrome-devtools
- chrome-devtools-protocol
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
- web-scraping
- text-extraction
- nlp
- html2text