A scalable web crawler framework for Java.
Repository Index
47,133 repositories indexed
1–6 of 6 · Most stars
jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety.
TranslatedA new-generation web crawler platform that defines crawler workflows graphically, so you can build a crawler without writing code.
Open Source Web Crawler for Java
Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.
- java
- elasticsearch
- full-text-search
- lucene
A scalable, mature and versatile web crawler based on Apache Storm
- web-crawler
- apache-storm
- distributed
- java