Incredibly fast crawler designed for OSINT.
Repository Index
55,640 visible repositories in the index
25–48 of 133 · Most stars
TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.
- webcrawler
- scrapy
- crawlab
- spiders-management
A scalable web crawler framework for Java.
GEO-first SEO skill for Claude Code. Comprehensive AI search optimization for any website — citability scoring, AI crawler analysis, brand authority, schema markup, platform-specific optimization, and PDF reports. If you want learn how to sell this to real businesses, check out the skool community
TranslatedA new-generation web crawler platform that defines crawler workflows graphically, so you can build a crawler without writing code.
TranslatedPython scripts. Simulated login to Zhihu, web crawlers, working with Excel, WeChat official accounts, remote power-on
UAParser.js: The Essential Web Development Tool for User-Agent Detection. Detect Browsers, OS, Devices, Bots, Apps, AI Crawlers, and more. Run in Browser or Node
- user-agent-parser
- browser-detection
- user-agent
- client-hints
TranslatedOneForAll is a powerful subdomain collection tool
- subdomain
- subdomain-scanner
- subdomain-enumeration
- subdomain-bruteforcing
TranslatedAV movie management system, avmoo, javbus, javlibrary crawler, online AV movie library, AV magnet link database, Japanese Adult Video Library, Adult Video Magnet Links - Japanese Adult Video Database
- javbus
- avmoo
- javlibrary
- spider
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.
- apify
- automation
- beautifulsoup
- crawler
TranslatedA self-media customer acquisition agent tailor-made for OPC/small, medium and micro enterprises
- mas
- multi-agent
- multi-agent-systems
- makemoney
TranslatedINFO-SPIDER is a crawler toolbox 🧰 that brings many data sources together in one place, designed to help users get their own data back safely and quickly. The tool's code is open source and the process is transparent. Supported data sources include GitHub, QQ Mail, NetEase Mail, Alibaba Mail, Sina Mail, Hotmail, Outlook Mail, JD.com, Taobao, Alipay, China Mobile, China Unicom, China Telecom, Zhihu, Bilibili, NetEase Cloud Music, QQ friends, QQ groups, generating a Moments (WeChat friend circle) photo album, browser browsing history, 12306, cnblogs (博客园), CSDN blog, OSChina blog, Jianshu.
List of libraries, tools and APIs for web scraping and data processing.
- web-scraping
- captcha-bypass
- captcha-recaptcha
- crawling
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
TranslatedPython API for JMComic | Provides a Python API for accessing 禁漫天堂 (JMComic), supporting both the web and mobile clients | 禁漫天堂 (JMComic) GitHub Actions downloader🚀
- 18comic
- crawler
- github-actions
- pypi
Ingest, parse, and optimize any data format ➡️ from documents to multimedia ➡️ for enhanced compatibility with GenAI frameworks
- ingestion-api
- ocr
- omniparser
- parse-server
Official Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.
- batch-processing
- claude
- content-extraction
- data-collection
Pholcus is a distributed high-concurrency crawler software written in pure golang
- spider
- crowler
The AI local media metadata crawler
- emby
- kodi
- plex
- python
One-stop Proxies Crawling and Aggregation Platform
- proxypool
A collection of awesome web crawler,spider in different languages
A Chrome DevTools Protocol driver for web automation and scraping.
- cdp
- chrome-headless
- chrome-devtools
- chrome-devtools-protocol
🔥 The fastest and powerful Python library for Instagram Private API 2026 with HikerAPI SaaS
- instagram-api
- instagram-client
- instagram-downloader
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
- web-scraping
- text-extraction
- nlp
- html2text