An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.
- archivespark
- spark-framework
- spark
- web-archiving
Repository Index
47,133 repositories indexed
1–5 of 5 · Most stars
The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-indexer' directory and code that exists in this repo is now only for reference. For support and issues of 'warc-indexer', please communicate with NetArchiveSuite.
Various examples of notebooks for working with web archives with the Archives Unleashed Toolkit, and derivatives generated by the Archives Unleashed Toolkit.
- notebooks
- pyspark-notebook
- juypter-notebook
- spark
An open-source toolkit for analyzing line-oriented JSON Twitter archives with Apache Spark.
- spark
- tweets
- twitter-data
- twitter-json