Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Fast and reliable browser automation for e2e testing, scraping, and stealth. Bypasses bot-detection with CDP Mode. Has integrations for pytest, Playwright, and MCP.
A scalable web crawler framework for Java.
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.
The All in One Framework to Build Undefeatable Scrapers
- anti-bot
- anti-detection
- cloudflare-bypass
- cloudflare-scrape
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
Extract any website's complete design system with one command. DTCG tokens, semantic+primitive+composite, MCP server for Claude Code/Cursor/Windsurf, multi-platform emitters (iOS SwiftUI, Android Compose, Flutter, WordPress), Tailwind v4, Figma variables, shadcn/ui, CSS health audit, WCAG remediation, Chrome extension. MIT, Playwright, Node 20+.
Apache Nutch is an extensible and scalable web crawler
Watch (parts of) webpages and get notified when something changes via e-mail, on your phone or via other means. Highly configurable.
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
Claim available free game promotions from the Epic Games Store.
- epicgames-freebies-claimer
- epicgames
- epic-games
- giveaway
A code for extracting best-selling items, search results, and currently available deals from Amazon using Python and Oxylabs E-Commerce Scraper API.
- amazon
- amazon-scraper
- api
- python
Open-source BigSet — self-hostable live datasets populated by TinyFish web agents
- open-source
- tinyfish
- bigset
Google, Naver multiprocess image web crawler (Selenium)
TranslatedIntelligent job search Agent (source-available, PolyForm Noncommercial license; commercial use requires separate written authorization)
A self hostable app that tracks prices and sends you notifications when prices match your preferences
- open-source
- price-comparison
- price-monitoring
- price-tracker
A blazing fast, async-first, undetectable webscraping/web automation framework based on ultrafunkamsterdam/nodriver. Now with Docker support!
- chrome
- chrome-devtools-protocol
- python
- webscraping
Collect structured responses from a ChatGPT scraper by sending a prompt with valid ChatGPT scraping API credentials. Enable live search, inject HTML context, and automate intelligent scraper ChatGPT workflows in just a few parameters.
A configurable and extensible PHP web spider
A scalable frontier for web crawlers
:rocket: Stealth - Secure, Peer-to-Peer, Private and Automateable Web Browser/Scraper/Proxy
Scalable Python web scraping scripts for +80 popular domains
Free desktop SEO crawler - open source alternative to Screaming Frog and similar tools. Crawl websites, analyze links, extract SEO data, and export results without subscription fees. Fully customizable and extensible!
Retrieves your activation data (activation_bytes) from Audible servers. Using https://github.com/inAudible-NG/tables project instead is recommended.