TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.
- webcrawler
- scrapy
- crawlab
- spiders-management
96,838 visible repositories in the index
1–12 of 12 · Most stars
TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.
scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place
Pholcus is a distributed high-concurrency crawler software written in pure golang
Distributed crawler powered by Headless Chrome
:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis
:zap: A distributed crawler for weibo, building with celery and requests.
DotnetSpider, a .NET standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现
一个简单、敏捷、分布式的支持SpringBoot的Java爬虫框架;An agile, distributed crawler framework.
Nov 20 2017 -- A distributed open source search engine and spider/crawler written in C/C++ for Linux on Intel/AMD. From gigablast dot com, which has binaries for download. See the README.md file at the very bottom of this page for instructions.
A scalable, mature and versatile web crawler based on Apache Storm