Web Crawler/Spider for NodeJS + server-side jQuery ;-)
Repository Index
132,241 visible repositories in the index
25–48 of 286 · Most stars
TranslatedSync articles to multiple content platforms with one click. Supports major platforms including Toutiao (今日头条), WordPress, Zhihu (知乎), Jianshu (简书), Juejin (掘金), CSDN, and typecho. Publish once, sync to multiple platforms. Free up personal productivity.
- chrome-extension
- chrome
- markdown
- multiplatform
An Efficient ProxyPool with Getter, Tester and Server
TranslatedHands-on 🐍 web crawlers for many kinds of websites and e-commerce data 🕷. Includes 🕸: Taobao products, WeChat official accounts, Dianping, Qichacha, job recruitment sites, Xianyu, Alibaba tasks, Cnblogs, Weibo, Baidu Tieba, Douban movies, Baotu (ibaotu.com), Quanjing (quanjing.com), Douban music, a provincial drug administration, Sohu News, machine learning text collection, FOFA asset collection, Autohome, National Bureau of Statistics, Baidu keyword indexed-page counts, spider pool directories, Toutiao (Today's Headlines), Douban movie reviews, Ctrip, Xiaomi App Store, Anjuke, Tujia homestays ❤️❤️❤️. WeChat crawler showcase project:
- python3
- crawler
- baidu-tieba
- taobao-spider
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
- high-availability
- scrapy
- ipproxy
- distributed
Dark Web OSINT Tool
- tor
- tor-network
- deepweb
- dark-web
Smart webfont compression and format conversion tool
- woff2
- woff
- ttf
- svg
:zap: A distributed crawler for weibo, building with celery and requests.
- weibospider
- data-analysis
- python3
- distributed-crawler
TranslatedToday's Hot List (今日热榜), an aggregator website that gets the hot headlines from major popular websites. Written in Go, it uses multiple goroutines to fetch information asynchronously and quickly. Preview: https://mo.fish
- hot
- golang
- spider
- today-s-hot-list
TranslatedSina Weibo crawler: uses Python to crawl Sina Weibo data and download Weibo images and Weibo videos
- crawler
- weibo-spider
TranslatedThis is a series of Java multithreading articles originally written and maintained by members of the RedSpider community.
Translated🇨🇳 Chinese edition of "Twenty Years of JavaScript"
- book
- javascript
- history
- browser
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
- crawler
- dotnetcore
- cross-platform
- csharp
持续维护的新浪微博采集工具🚀🚀🚀
- scrapy
- python
- weibospider
Git All the Payloads! A collection of web attack payloads.
- payload
- payloads
- xss
- sqli
新闻网页正文通用抽取器 Beta 版.
- python3
- webcrawler
- webspider
🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单,功能强大的Python爬虫框架。内置AirSpider、Spider、TaskSpider、BatchSpider四种爬虫解决不同场景的需求。且支持断点续爬、监控报警、浏览器渲染、海量数据去重等功能。更有功能强大的爬虫管理系统feaplat为其提供方便的部署及调度
抖音逆向,抖音爬虫,抖音全部api、私信、直播间监听
Every web site provides APIs.
A DNS meta-query spider that enumerates DNS records, and subdomains.
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
- scrapy
- distributed
- webspider
- scrapyd
微信公众号文章的爬虫
《我用爬虫一天时间“偷了”知乎一百万用户,只为证明PHP是世界上最好的语言 》所使用的程序
- php
- phpspider
Python爬虫实战 - 模拟登陆各大网站 包含但不限于:滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝,如果喜欢请start ❤️
- scrapy
- python
- crawl
- crawler