🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
🧡 Everything is RSSible
🕵️♂️ Collect a dossier on a person by username from 6K websites
- osint
- social-network
- identification
- socmint
Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring— MCP included! all for free or enjoy our SaaS plan!
- website-monitor
- website-monitoring
- change-detection
- monitoring
Python scraper based on AI
TranslatedSelf-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
TranslatedDouyin / TikTok platform tool for downloading works and collecting data
Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection
Download pictures (or videos) along with their captions and other metadata from Instagram.
- instagram
- instagram-client
- instagram-feed
- downloader
TranslatedXiaohongshu (XiaoHongShu, RedNote) link extraction / post collection tool
Fast and reliable browser automation for e2e testing, scraping, and stealth. Bypasses bot-detection with CDP Mode. Has integrations for pytest, Playwright, and MCP.
A scalable web crawler framework for Java.
A standalone version of the readability lib
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.
Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.
Python Script to download hundreds of images from 'Google Images'. It is a ready-to-run code!
- python
- python-script
- image-download
- google-images
This is a python API which allows you to get the transcript/subtitles for a given YouTube video. It also works for automatically generated subtitles and it does not require an API key nor a headless browser, like other selenium based solutions do!
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
TranslatedGet the real streaming URLs (live stream sources) and danmaku (live comments) of 58 live streaming platforms, including Douyu, Huya, Bilibili, Douyin, and Kuaishou. The live stream sources can be played in players such as PotPlayer and flv.js.
- douyu
- huya
- bilibili
- python
The AI local media metadata crawler
Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.
Turn any webpage into structured data using LLMs
The Unofficial TikTok API Wrapper In Python
- tiktok-api
- python
- api
- trending