Repository Index

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

• spark
• acid
• big-data
• analytics

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

• spark
• hive
• sql
• thrift

An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.

• archivespark
• spark-framework
• spark
• web-archiving

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

• spark
• hadoop
• webarchives
• analysis

A library that brings useful functions from various modern database management systems to Apache Spark

• spark
• postgres
• presto
• hive

Web application for distributed compute analysis of Archive-It web archive collections.

Internet Archive's Sparkling Data Processing Library

An open-source toolkit for analyzing line-oriented JSON Twitter archives with Apache Spark.

• spark
• tweets
• twitter-data
• twitter-json