A scalable web crawler framework for Java.
Repository Index
72,362 visible repositories in the index
1–10 of 10 · Most stars
TranslatedA new-generation web crawler platform that defines crawler workflows graphically, so you can build a crawler without writing code.
Open Source Web Crawler for Java
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
- java
- webcrawling
- warc
- heritrix
Apache Nutch is an extensible and scalable web crawler
- java
- nutch
- web-crawler
- crawling
WebCollector is an open source web crawler framework based on Java.It provides some simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.
Easy to use lightweight web crawler（易用的轻量化网络爬虫）
- gecco
- crawler
- jsoup
- fastjson
Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.
- java
- elasticsearch
- full-text-search
- lucene
A scalable, mature and versatile web crawler based on Apache Storm
- web-crawler
- apache-storm
- distributed
- java
Java API For Chrome and Firefox