Repository Index

Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.

• agent-infrastructure
• ai-agent
• ai-search
• automation

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

• crawler
• crawling
• crawling-python
• playwright

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

• web-scraping
• web-crawling
• npm
• headless-chrome

Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust

• browser
• rust
• servo
• browser-automation

Turn any website into a structured API. Extract, automate, search and monitor the web.

• automation
• no-code
• scraper
• web-scraper

TranslatedA multi-task real-time/scheduled monitoring and intelligent analysis system for Xianyu (闲鱼), built with Playwright and AI, and equipped with a full-featured back-end management UI. It helps users find the products they like among the vast number of items on Xianyu.

• ai
• playwright
• automation
• gemini

Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection

• ai-tools
• automation
• claude-ai
• claude-skills

Browser automation and testing without limits. SeleniumBase's CDP Mode bypasses bot-detection with ease. Check out the examples!

• python
• selenium-python
• e2e-testing
• seleniumbase

A Chrome DevTools Protocol driver for web automation and scraping.

• cdp
• chrome-headless
• chrome-devtools
• chrome-devtools-protocol

A Multi-User Selfhosted price tracker for Amazon, Aliexpress, ebay and many more along with custom stores. Get notified when price matches one or more criterias set by the user

• automation
• ecommerce
• filamentphp
• laravel

A AI Browser run any site JavaScript from Python with the real Firefox engine. Headless SpiderMonkey + Necko for automation and scraping.

• anti-bot
• automation
• firefox
• headless-browser

Self-hosted web scraper, crawler and document parser for AI agents: URLs, PDFs with OCR and Office files to clean Markdown/JSON. Firecrawl- and Jina-compatible API, MCP server, agent endpoint, Python/Node SDKs. Single Rust binary.

• ai-agents
• browser-automation
• data-extraction
• firecrawl