70,736 visible repositories in the index
1–24 of 73 · Most stars
🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
🕵️♂️ Collect a dossier on a person by username from 6K websites
Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring— MCP included! all for free or enjoy our SaaS plan!
newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
Download pictures (or videos) along with their captions and other metadata from Instagram.
A scalable web crawler framework for Java.
This is a python API which allows you to get the transcript/subtitles for a given YouTube video. It also works for automatically generated subtitles and it does not require an API key nor a headless browser, like other selenium based solutions do!
TranslatedXiaohongshu (Little Red Book) crawler data collection, Xiaohongshu reverse engineering, private messages, live streaming, full-domain operations solution for Xiaohongshu
TranslatedINFO-SPIDER is a crawler toolbox 🧰 that brings many data sources together in one place, designed to help users get their own data back safely and quickly. The tool's code is open source and the process is transparent. Supported data sources include GitHub, QQ Mail, NetEase Mail, Alibaba Mail, Sina Mail, Hotmail, Outlook Mail, JD.com, Taobao, Alipay, China Mobile, China Unicom, China Telecom, Zhihu, Bilibili, NetEase Cloud Music, QQ friends, QQ groups, generating a Moments (WeChat friend circle) photo album, browser browsing history, 12306, cnblogs (博客园), CSDN blog, OSChina blog, Jianshu.
The AI local media metadata crawler
Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.
The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.
Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application
Extract any website's complete design system with one command. DTCG tokens, semantic+primitive+composite, MCP server for Claude Code/Cursor/Windsurf, multi-platform emitters (iOS SwiftUI, Android Compose, Flutter, WordPress), Tailwind v4, Figma variables, shadcn/ui, CSS health audit, WCAG remediation, Chrome extension. MIT, Playwright, Node 20+.
Successor of Undetected-Chromedriver. Providing a blazing fast framework for web automation, webscraping, bots and any other creative ideas which are normally hindered by annoying anti bot systems like Captcha / CloudFlare / Imperva / hCaptcha
An agentic extractor for messaging platforms. Noria receives messages & URLs via chat; scrapes web data to evaluate the content against custom scoring matrices (e.g., jobs, scholarships); and sends a structured summary with a calculated match score directly to the assigned messenger number.
job-ops: DevOps principles applied to job hunting. A self-hosted pipeline to track, analyze, and assist your application process
🔮 A Node.js scraper for humans.
Automatically crawls proxy nodes on the public internet, de-duplicates and tests for usability and then provides a list of nodes
Intelligent proxy pool for Humans™ to extract content from the internet and build your own Large Language Models in this new AI era
All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
抖音逆向，抖音爬虫，抖音全部api、私信、直播间监听