Repository Index

A scalable, mature and versatile web crawler based on Apache Storm

• web-crawler
• apache-storm
• distributed
• java

Turn any website into clean, structured data: Markdown, metadata and links from one page or a whole site, in every major language.

• crawling
• csharp
• elixir
• ffi