Generate and download e-books from online sources.
- lightnovel
- termux
- web-scraper
- console-app
Repository Index
72,862 visible repositories in the index
121–144 of 227 · Most stars
:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
🙌Easily download all the videos from TikTok(amemv).下载指定的 抖音（Douyin） 号的视频,抖音爬虫
🔥 Visual workflow builder for AI agents powered by Firecrawl - drag-and-drop web scraping pipelines with real-time execution
A curated list of awesome puppeteer resources.
- puppeteer
- headless-chrome
- awesome
- awesome-list
Easy to use lightweight web crawler（易用的轻量化网络爬虫）
- gecco
- crawler
- jsoup
- fastjson
Production ready MCP server with real-time search, extract, map & crawl.
Leaked GPTs Prompts Bypass the 25 message limit or to try out GPTs without a Plus subscription.
- ai
- awesome
- awesome-list
- gpts
Web Scraping Framework
- web-scraping
- http-client
- framework
- python
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
ScopeSentry-Cyberspace mapping, subdomain enumeration, port scanning, sensitive information discovery, vulnerability scanning, distributed nodes
- bugbounty
- bugbounty-tool
- crawler
- dirscanner
Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator
- scraper
- sina
- corpus
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
- c-sharp
- abot
- abot-nuget
- crawler
Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.
- golang
- go
- cloning
- crawler
dhtcrawler is a DHT crawler written in erlang. It can join a DHT network and crawl many P2P torrents. The program save all torrent info into database and provide an http interface to search a torrent by a keyword
📅🇨🇳中国法定节假日数据 自动每日抓取国务院公告
- data
- natural-language-processing
- crawling
- holiday
Web Crawling and RAG Capabilities for AI Agents and AI Coding Assistants
🏳️🌈 Media downloader from any sites, including Twitter, Reddit, Instagram, BlueSky, TikTok, Threads, Facebook, OnlyFans, YouTube, Pinterest, PornHub, XHamster, XVIDEOS, ThisVid etc.
Transform Web Content into LLM-Ready Data
- crawl4ai
- crawler
- crawling-python
- html2markdown
Internet search engine for text-oriented websites. Indexing the small, old and weird web.
- search-engine
- no-cloud
- small-web
- internet-search
基于小红书 Web 端进行的请求封装。https://reajason.github.io/xhs/
- crawl
- python
- xhs
vulnx 🕷️ an intelligent Bot, Shell can achieve automatic injection, and help researchers detect security vulnerabilities CMS system. It can perform a quick CMS security detection, information collection (including sub-domain name, ip address, country information, organizational information and time zone, etc.) and vulnerability scanning.
- crawler
- information-gathering
- cms-detector
- cloudflare-detection
Rust actor framework