A complete and versatile web scraper.
Repository Index
91,851 visible repositories in the index
1–3 of 3 · Most stars
A scalable, mature and versatile web crawler based on Apache Storm
- web-crawler
- apache-storm
- distributed
- java
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.