Repository Index

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

• ai
• crawler
• markdown
• scraper

Build Agentic workflows, RAG pipelines, with rich AI model and tool support on one collaborative workspace. Deploy on cloud, VPC, or self-hosted, so teams move from prototype to production without rebuilding the stack.

• ai
• gpt
• llm
• openai

Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.

• agent-infrastructure
• ai-agent
• ai-search
• automation

Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.

• playwright
• testing
• automation
• webkit

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

• crawler
• crawling
• crawling-python
• playwright

JavaScript API for Chrome and Firefox

• headless-chrome
• testing
• web
• developer-tools

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

• ai
• ai-agents
• crawler
• data-extraction

Your lightweight open-source personal agent.

• ai-agent
• ai-agents
• openclaw
• agent-framework

🧡 Everything is RSSible

• rss
• twitter
• weibo
• bilibili

📨 The ultimate agentic social media scheduling tool 🤖

• typescript
• nextjs
• redis
• open-source

TranslatedAn MCP-based chatbot

• chatbot
• esp32
• mcp

Python scraper based on AI

• scraping
• scraping-python
• llm
• web-crawler

The fast, flexible, and elegant library for parsing and manipulating HTML and XML.

• cheerio
• jquery
• htmlparser2
• dom

Sim is the collaborative workspace to build, deploy, and monitor AI agents and workflows. Used by 100,000+ builders.

• agentic-workflow
• agents
• ai
• nextjs

🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

• pocket
• wget
• browser-bookmarks
• pinboard

Let's use AI to Earn!

• auto-publish
• douyin
• electron-app
• electron-react

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

• web-scraping
• web-crawling
• npm
• headless-chrome

The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more.

• agents
• ai
• playwright
• ai-agents

Python ProxyPool for web spider

• crawler
• proxy
• spider
• redis

Self-Hosting Guide. Learn all about locally hosting (on premises & private web servers) and managing software applications by yourself or your organization. Including Cloud, LLMs, WireGuard, Automation, Home Assistant, and Networking.

• self-hosted
• selfhosted
• home-assistant
• wireguard

Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust

• browser
• rust
• servo
• browser-automation

✨ AI Spreadsheet for Business

• airtable
• airtable-alternative
• database
• no-code

The premier source of truth powering network automation. Open source under Apache 2. Try NetBox Cloud free: https://netboxlabs.com/products/free-netbox-cloud/

• dcim
• ipam
• netbox
• network-automation

A browser extension for automating your browser by connecting blocks

• vue
• chrome-extension
• automation
• browser-extension