Async Python 3.6+ web scraping micro-framework based on asyncio
- asyncio
- aiohttp
- asyncio-spider
- crawler
132,241 visible repositories in the index
97–120 of 286 · Most stars
Async Python 3.6+ web scraping micro-framework based on asyncio
微博爬虫及配套工具箱,微博用户、话题、评论采集一网打尽。图片下载、情感分析,地理位置、关系网络、spammer 机器人识别等功能应有尽有。Docs:https://buyixiao.github.io/blog/weibo-super-spider.html 配套可视化网站:https://buyixiao.github.io/blog/one-stop-weibo-visualization.html
HeapDump敏感信息提取工具
python爬虫,目前库存:网易云音乐歌曲爬取,B站视频爬取,知乎问答爬取,壁纸爬取,xvideos视频爬取,有声书爬取,微博爬虫,安居客信息爬取+数据可视化,哔哩哔哩视频封面提取器,ip代理池封装,知乎百万级用户爬虫+数据分析,github用户爬虫
抖音爬虫——采集账号主页、喜欢、收藏、音乐原声、话题、搜索、合集、作品、关注、粉丝等公开数据。
The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
Nov 20 2017 -- A distributed open source search engine and spider/crawler written in C/C++ for Linux on Intel/AMD. From gigablast dot com, which has binaries for download. See the README.md file at the very bottom of this page for instructions.
Anemone web-spider framework
微信爬虫,获取文章内容、阅读量、点赞量、评论等,获取公众号所有历史文章链接。
一款信息泄漏利用工具,适用于.git/.svn/.DS_Store泄漏和目录列出
Encryptr is a zero-knowledge cloud-based password manager / e-wallet powered by Crypton
This is a lightweight, headless browser purpose-built for AI agents (such as Codex or Claude Code) to perform browser automation tasks; it is designed to navigate through various web protection systems, login processes, and reverse-engineering scenarios
手撕包菜网站
小红书数据采集、网站图片、视频资源批量下载工具,颜值超高的数据采集工具(批量下载,视频提取,图片)Telegram:https://t.me/+ZtLSwuIKTo44MDY1
Nodejs实现的一个磁力链接爬虫 https://findit.keenwon.com (原域名http://findit.so )
Spider entire networks for juicy files sitting on SMB shares. Search filenames or file content - regex supported!
:cloud: The SCSS framework for the modern web.
浏览过的精彩逆向文章汇总,值得一看
📰 Diários oficiais brasileiros acessíveis a todos | 📰 Brazilian government gazettes, accessible to everyone.
Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome :star:
一个主要运行在 𝐞𝐥𝐞𝐜𝐕𝟐𝐏 或 𝐪𝐢𝐧𝐠𝐥𝐨𝐧𝐠 等定时面板,同时支持系统运行环境的签到项目(环境:𝑷𝒚𝒕𝒉𝒐𝒏 3.8+ / 𝑵𝒐𝒅𝒆.𝒋𝒔 10+ / 𝑩𝒂𝒔𝒉 4+ / 𝑶𝒑𝒆𝒏𝑱𝑫𝑲8 / 𝑷𝒆𝒓𝒍5)
K 哥爬虫代码分享,JS 逆向,爬虫进阶。关注公众号:K哥爬虫
The repository host gives no description.
A configurable and extensible PHP web spider