Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Repository Index
55,640 visible repositories in the index
1–18 of 18 · Most stars
🔥 Clone and recreate any website as a modern React app in seconds
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
- web-scraping
- web-crawling
- npm
- headless-chrome
Crawl a site to generate knowledge files to create your own custom GPT from a URL
Turn any website into a structured API. Extract, automate, search and monitor the web.
- automation
- no-code
- scraper
- web-scraper
The Apify MCP server enables your AI agents to extract data from social media, search engines, maps, e-commerce sites, or any other website using thousands of ready-made scrapers, crawlers, and automation tools available on the Apify Store.
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
An open source deep research clone. AI Agent that reasons large amounts of web data extracted with Firecrawl
The All in One Framework to Build Undefeatable Scrapers
- anti-bot
- anti-detection
- cloudflare-bypass
- cloudflare-scrape
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
TranslatedSelf-hosted web and AI crawler analytics for Generative Engine Optimization (GEO). Track visitors, AI search referrals, crawl logs, content coverage, and conversions. Jointly published by 猎看科技 (Liekan Technology, "hunt and see technology") · 北京日报社智源GEO (Beijing Daily Zhiyuan GEO, "Beijing Daily Press Zhiyuan GEO"). MIT licensed.
- ai-analytics
- geo
- mit
- postgresql
Scout: the open-source browser agent, by Outcrawl. Mirrored automatically from Outcrawl's own repository.
Xiryn — the open-source personal agent. Does everything Muse, Grok Bot and Dots do, on your own computer: each agent gets its own Linux desktop in Docker, any model with your keys, an encrypted vault, and control from Telegram.
A Payload CMS 3 plugin for Auth.js/NextAuth 5
- payload-plugin
- payloadcms
- authjs
- next-auth
:mag: Data pipeline for crawling PDFs from the Web and transforming their contents into structured data using AWS textract. Built with AWS CDK + TypeScript
a dungeon crawler written in TypeScript using React and svg
Practical pinned gist which show your latest medium status 📌
- awesome-pinned-gists
- medium-stats
- github-gists
- github-action
The repository host gives no description.