An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.
- archivespark
- spark-framework
- spark
- web-archiving
89,572 visible repositories in the index
1–4 of 4 · Most stars
An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.
Distributed batch data processing framework
A Splitable Hadoop InputFormat for Concatenated GZIP Files and *.(w)arc.gz
Partition (W)ARC Files by MIME Type and Year