Anemone web-spider framework
Repository Index
47,133 repositories indexed
1–3 of 3 · Most stars
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Voight-Kampff is a Ruby gem that detects bots, spiders, crawlers and replicants