A scalable web crawler framework for Java.
Repository Index
132,241 visible repositories in the index
1–16 of 16 · Most stars
TranslatedA new-generation web crawler platform that defines crawler workflows graphically, so you can build a crawler without writing code.
Open Source Web Crawler for Java
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
- java
- webcrawling
- warc
- heritrix
Apache Nutch is an extensible and scalable web crawler
- java
- nutch
- web-crawler
- crawling
WebCollector is an open source web crawler framework based on Java.It provides some simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.
Easy to use lightweight web crawler(易用的轻量化网络爬虫)
- gecco
- crawler
- jsoup
- fastjson
NewPipe's core library for extracting data from streaming sites
一个简单、敏捷、分布式的支持SpringBoot的Java爬虫框架;An agile, distributed crawler framework.
Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.
- java
- elasticsearch
- full-text-search
- lucene
A scalable, mature and versatile web crawler based on Apache Storm
- web-crawler
- apache-storm
- distributed
- java
zhihu-crawler是一个基于Java的高性能、支持免费http代理池、支持横向扩展、分布式爬虫项目
Java API For Chrome and Firefox
A lightweight web crawler framework.(Java爬虫框架)
NetDiscovery 是一款基于 Vert.x、RxJava 2 等框架实现的通用爬虫框架/中间件。
Crawler Microservice for the YaCy Grid