TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.
- webcrawler
- scrapy
- crawlab
- spiders-management
96,838 visible repositories in the index
1–4 of 4 · Most stars
TranslatedDistributed web crawler admin platform for spiders management regardless of languages and frameworks.
scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place
Pholcus is a distributed high-concurrency crawler software written in pure golang
A distributed job server built specifically for queuing and executing heavy SQL read jobs asynchronously. Separate out reporting layer from apps. MySQL, Postgres, ClickHouse.