47,133 repositories indexed
1–22 of 22 · Most stars
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Scrapy, a fast high-level web crawling & scraping framework for Python.
Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 50+ tests passed.
Python scraper based on AI
🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...
TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection
Pythonic HTML Parsing for Humans™
Browser automation and testing without limits. SeleniumBase's CDP Mode bypasses bot-detection with ease. Check out the examples!
Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.
Lighter web automation with Python
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
A Python module to bypass Cloudflare's anti-bot page.
Playwright that anti-bots cannot see, so no captchas: same API, anti-detect stealth headless Firefox, undetected fingerprint, bypass bot detection, Python web scraping and browser automation.
Scrape Twitter for Tweets
Async Python 3.6+ web scraping micro-framework based on asyncio
A chat bot for Slack (https://slack.com).
:scissors: High performance, multi-threaded image scraper
📣 A tiny cron job that issues a tweet with Algorand Block proposal stats on daily basis
Intelligent web page comparison tool with Wayback Machine support and visual regression testing