DecryptLogin: APIs for loginning some websites by using requests.
- requests
- login
- python3
- spider
132,241 visible repositories in the index
73–96 of 248 · Most stars
DecryptLogin: APIs for loginning some websites by using requests.
https://spatie.be/docs/crawler
Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.
Generate and download e-books from online sources.
:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
🙌Easily download all the videos from TikTok(amemv).下载指定的 抖音(Douyin) 号的视频,抖音爬虫
Easy to use lightweight web crawler(易用的轻量化网络爬虫)
Web Scraping Framework
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
dhtcrawler is a DHT crawler written in erlang. It can join a DHT network and crawl many P2P torrents. The program save all torrent info into database and provide an http interface to search a torrent by a keyword
Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.
Transform Web Content into LLM-Ready Data
vulnx 🕷️ an intelligent Bot, Shell can achieve automatic injection, and help researchers detect security vulnerabilities CMS system. It can perform a quick CMS security detection, information collection (including sub-domain name, ip address, country information, organizational information and time zone, etc.) and vulnerability scanning.
Real time google image scraper. The process is automated by sending HTTP requests to retrieve image data which is then parsed and saved.
蓝天采集器是一款开源免费的爬虫系统,仅需点选编辑规则即可采集数据,可运行在本地、虚拟主机或云服务器中,几乎能采集所有类型的网页,无缝对接各类CMS建站程序,免登录实时发布数据,全自动无需人工干预!是网页大数据采集软件中完全跨平台的云端爬虫系统
NewPipe's core library for extracting data from streaming sites
Polite, slim and concurrent web crawler.
Web crawling framework based on asyncio.
Find web directories without bruteforce
一个简单、敏捷、分布式的支持SpringBoot的Java爬虫框架;An agile, distributed crawler framework.
爬虫案例合集。包括但不限于《淘宝、京东、天猫、豆瓣、抖音、快手、微博、微信、阿里、头条、pdd、优酷、爱奇艺、携程、12306、58、搜狐、各种指数、维普万方、Zlibraty、Oalib、小说、招标网、采购网、小红书、大众点评、推特、脉脉、知乎》