Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Repository Index
78,351 visible repositories in the index
1–24 of 36 · Most stars
Clone any website with one command using AI coding agents
The fast, flexible, and elegant library for parsing and manipulating HTML and XML.
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
- web-scraping
- web-crawling
- npm
- headless-chrome
The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and more.
An AI-powered research assistant that performs iterative, deep research on any topic by combining search engines, web scraping, and large language models. The goal of this repo is to provide the simplest implementation of a deep research agent - e.g. an agent that can refine its research direction overtime and deep dive into a topic.
Turn any website into a structured API. Extract, automate, search and monitor the web.
- automation
- no-code
- scraper
- web-scraper
The Apify MCP server enables your AI agents to extract data from social media, search engines, maps, e-commerce sites, or any other website using thousands of ready-made scrapers, crawlers, and automation tools available on the Apify Store.
Open-source AI agent for web automation. Use your real, logged-in browser to automate daily tasks.
Turn any webpage into structured data using LLMs
NodeJS Instagram private API SDK. Written in TypeScript.
- instagram-api
- instagram-client
- instagram-feed
Official Bright Data CLI - scrape, search, and extract structured web data directly from your terminal.
📈 Get real-time stocks from TradingView
- tradingview
- tradingview-api
- realtime-stocks
- stocks
The All in One Framework to Build Undefeatable Scrapers
- anti-bot
- anti-detection
- cloudflare-bypass
- cloudflare-scrape
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
TikTok Scraper. Download video posts, collect user/trend/hashtag/music feed metadata, sign URL and etc.
Undetected version of the Playwright testing and automation library.
- automation
- bot
- bots
- botting
Datascraper that is currently adult focused - Updated regularly
- onlyfans
- datascraping
- scraper
- archive
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
- aitools
- crawl
- scrape
- serp
Advanced Privacy Browser Core with Unified Fingerprint Defense: Cloudflare, Akamai, Kasada, Shape, DataDome, PerimeterX, hCaptcha, FunCaptcha, Imperva, reCAPTCHA, ThreatMetrix, Adscore
- akamai
- cloudflare
- incapsula
- perimeterx
Browser fingerprinting tools for anonymizing your scrapers. Developed by Apify.
- fingerprinting
- playwright
- puppeteer
- scraping
🔥 Visual workflow builder for AI agents powered by Firecrawl - drag-and-drop web scraping pipelines with real-time execution
Meridian cuts through news noise by scraping hundreds of sources, analyzing stories with AI, and delivering concise, personalized daily briefs.
The undetected self-hosted browser automation platform. Powered by Camoufox (Firefox) for 0% detection rates. Built for speed, privacy, and scalability.
- automation
- browser-automation
- browserless
- chromedriver