Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Repository Index
47,133 repositories indexed
1–24 of 27 · Most stars
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Scrapy, a fast high-level web crawling & scraping framework for Python.
🕵️♂️ Collect a dossier on a person by username from 6K websites
Python scraper based on AI
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Elegant Scraper and Crawler Framework for Golang
Pythonic HTML Parsing for Humans™
Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
Stealth headless browser for AI agents — bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.
A scalable web crawler framework for Java.
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
Mechanize is a ruby library that makes automated web interaction easy.
A browser testing and web crawling library for PHP and Symfony
A curated list of awesome puppeteer resources.
Playwright that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Python web scraping and browser automation.
Get info from any web service or page
:rocket: An open source alternative to searx which provides a modern-looking :sparkles:, lightning-fast :zap:, privacy respecting :disguised_face:, secure :lock: meta search engine
Tools for various online judges. Downloading sample cases, generating additional test cases, testing your code, and submitting it.
Crawly, a high-level web crawling & scraping framework for Elixir.
:scissors: High performance, multi-threaded image scraper
Extract structured data from web sites. Web sites scraping.
Functional HTML scraping and rewriting with CSS in OCaml
Scrape any website, article or RSS/Atom Feed with ease!