Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.
- scraper
- twitter-api
- twitter-bot
Repository Index
72,862 visible repositories in the index
73–96 of 135 · Most stars
owllook-小说搜索引擎
A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.
- search-engine
- python
- scraping
- crawler
DecryptLogin: APIs for loginning some websites by using requests.
- requests
- login
- python3
- spider
Foundational low latency web data collecting in Rust
https://spatie.be/docs/crawler
- php
- crawler
- guzzle
- concurrency
Find way more from the Wayback Machine, Common Crawl, Alien Vault OTX, URLScan, VirusTotal, GhostArchive & Intelligence X!
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.
Generate and download e-books from online sources.
- lightnovel
- termux
- web-scraper
- console-app
:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
🙌Easily download all the videos from TikTok(amemv).下载指定的 抖音（Douyin） 号的视频,抖音爬虫
🔥 Visual workflow builder for AI agents powered by Firecrawl - drag-and-drop web scraping pipelines with real-time execution
Easy to use lightweight web crawler（易用的轻量化网络爬虫）
- gecco
- crawler
- jsoup
- fastjson
Web Scraping Framework
- web-scraping
- http-client
- framework
- python
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator
- scraper
- sina
- corpus
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
- c-sharp
- abot
- abot-nuget
- crawler
Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.
- golang
- go
- cloning
- crawler
dhtcrawler is a DHT crawler written in erlang. It can join a DHT network and crawl many P2P torrents. The program save all torrent info into database and provide an http interface to search a torrent by a keyword
Transform Web Content into LLM-Ready Data
- crawl4ai
- crawler
- crawling-python
- html2markdown
基于小红书 Web 端进行的请求封装。https://reajason.github.io/xhs/
- crawl
- python
- xhs
vulnx 🕷️ an intelligent Bot, Shell can achieve automatic injection, and help researchers detect security vulnerabilities CMS system. It can perform a quick CMS security detection, information collection (including sub-domain name, ip address, country information, organizational information and time zone, etc.) and vulnerability scanning.
- crawler
- information-gathering
- cms-detector
- cloudflare-detection