Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Repository Index
72,362 visible repositories in the index
1–24 of 25 · Most stars
🔥 Clone and recreate any website as a modern React app in seconds
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
- web-scraping
- web-crawling
- npm
- headless-chrome
Crawl a site to generate knowledge files to create your own custom GPT from a URL
Turn any website into a structured API. Extract, automate, search and monitor the web.
- automation
- no-code
- scraper
- web-scraper
The Apify MCP server enables your AI agents to extract data from social media, search engines, maps, e-commerce sites, or any other website using thousands of ready-made scrapers, crawlers, and automation tools available on the Apify Store.
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
An open source deep research clone. AI Agent that reasons large amounts of web data extracted with Firecrawl
The All in One Framework to Build Undefeatable Scrapers
- anti-bot
- anti-detection
- cloudflare-bypass
- cloudflare-scrape
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
Exa MCP for web search and web crawling!
- code-search
- codesearch
- crawling
- mcp
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
- aitools
- crawl
- scrape
- serp
Article publishing platform that automatically distributes your articles to various media channels
🔥 Visual workflow builder for AI agents powered by Firecrawl - drag-and-drop web scraping pipelines with real-time execution
Production ready MCP server with real-time search, extract, map & crawl.
Transform Web Content into LLM-Ready Data
- crawl4ai
- crawler
- crawling-python
- html2markdown
Completely free, private, UI based Tech Documentation MCP server. Designed for coders and software developers in mind. Easily integrate into Cursor, Windsurf, Cline, Roo Code, Claude Desktop App
- crawl4ai
- documentation
- llm
- playwright
TranslatedSelf-hosted web and AI crawler analytics for Generative Engine Optimization (GEO). Track visitors, AI search referrals, crawl logs, content coverage, and conversions. Jointly published by 猎看科技 (Liekan Technology, "hunt and see technology") · 北京日报社智源GEO (Beijing Daily Zhiyuan GEO, "Beijing Daily Press Zhiyuan GEO"). MIT licensed.
- ai-analytics
- geo
- mit
- postgresql
Scout: the open-source browser agent, by Outcrawl. Mirrored automatically from Outcrawl's own repository.
Xiryn — the open-source personal agent. Does everything Muse, Grok Bot and Dots do, on your own computer: each agent gets its own Linux desktop in Docker, any model with your keys, an encrypted vault, and control from Telegram.
A Payload CMS 3 plugin for Auth.js/NextAuth 5
- payload-plugin
- payloadcms
- authjs
- next-auth
:mag: Data pipeline for crawling PDFs from the Web and transforming their contents into structured data using AWS textract. Built with AWS CDK + TypeScript
a dungeon crawler written in TypeScript using React and svg
Practical pinned gist which show your latest medium status 📌
- awesome-pinned-gists
- medium-stats
- github-gists
- github-action