An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.
- archivespark
- spark-framework
- spark
- web-archiving
Repository Index
47,133 repositories indexed
1–5 of 5 · Most stars
The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
A robust web archive analytics toolkit
- python
- web
- warc
- bigdata
A Splitable Hadoop InputFormat for Concatenated GZIP Files and *.(w)arc.gz
Partition (W)ARC Files by MIME Type and Year
- web-archiving
- webarchive
- hadoop
- warc