Repository Index

Apache Spark - A unified analytics engine for large-scale data processing

• python
• scala
• r
• java

CMAK is a tool for managing Apache Kafka clusters

• kafka
• scala
• cluster-management
• big-data

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

• spark
• acid
• big-data
• analytics

The leader in Customer Data Infrastructure

• snowplow
• analytics
• data
• data-pipeline

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

• dataquality
• spark
• unit-testing
• scala

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

• spark
• hive
• sql
• thrift

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

• apache-spark
• big-data
• connected-components
• dataframe

Sparkling Water provides H2O functionality inside Spark cluster

• h2o
• spark
• machine-learning
• integration

Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.

• livy
• bigdata
• spark
• apachelivy

Essential Spark extensions and helper methods ✨😲

• spark
• dataframe

Lightweight real-time big data streaming engine over Apache Pekko

• akka
• stream-processing
• bigdata
• scala

Low-code tool for automating actions on real time data | Stream processing for the users.

• flink
• flink-kafka
• apache-flink
• gui

The Apache Spark - Apache HBase Connector is a library to support Spark accessing HBase table as external data source or sink.

Neo4j Connector for Apache Spark, which provides bi-directional read/write access to Neo4j from Spark, using the Spark DataSource APIs

• spark
• neo4j-connector
• cypher
• neo4j-driver

An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.

• archivespark
• spark-framework
• spark
• web-archiving

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

• spark
• hadoop
• webarchives
• analysis

A library that brings useful functions from various modern database management systems to Apache Spark

• spark
• postgres
• presto
• hive

An open-source toolkit for analyzing line-oriented JSON Twitter archives with Apache Spark.

• spark
• tweets
• twitter-data
• twitter-json