A scalable web crawler framework for Java.
Repository Index
72,362 visible repositories in the index
1–18 of 18 · Most stars
TranslatedA new-generation web crawler platform that defines crawler workflows graphically, so you can build a crawler without writing code.
Shattered Pixel Dungeon is an open-source traditional roguelike dungeon crawler with randomized levels and enemies, and hundreds of items to collect and use. It's based on the source code of Pixel Dungeon, by Watabou.
Open Source Web Crawler for Java
Scrape Google AI Mode responses without blocks on a large scale.
- ai-mode
- google-ai
- google-ai-mode
- proxy-scrape
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
- java
- webcrawling
- warc
- heritrix
Apache Nutch is an extensible and scalable web crawler
- java
- nutch
- web-crawler
- crawling
WebCollector is an open source web crawler framework based on Java.It provides some simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.
Easy to use lightweight web crawler（易用的轻量化网络爬虫）
- gecco
- crawler
- jsoup
- fastjson
Internet search engine for text-oriented websites. Indexing the small, old and weird web.
- search-engine
- no-cloud
- small-web
- internet-search
Free database schema discovery and comprehension tool
- schemacrawler
- jdbc
- database-diagrams
- schema
Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/RAG & semantic search. Apache-2.0.
- java
- elasticsearch
- full-text-search
- lucene
A scalable, mature and versatile web crawler based on Apache Storm
- web-crawler
- apache-storm
- distributed
- java
Java API For Chrome and Firefox
Converts HTTrack crawls to WARC files
- web-archiving
a weirdly addictive arcade-style android game, where you fling fruit at a wall.
"wannabe tempest" for android
"wannabe tempest" -- a tempest clone in java