Repository Index

Apache Spark - A unified analytics engine for large-scale data processing

• python
• scala
• r
• java

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

• spark
• acid
• big-data
• analytics

Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark

• spark
• pyspark
• azure
• scala

State of the Art Natural Language Processing

• nlp
• natural-language-processing
• spark
• pyspark

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

• dataquality
• spark
• unit-testing
• scala

REST job server for Apache Spark

• spark
• rest-api
• spark-jobserver
• scala

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

• spark
• hive
• sql
• thrift

Apache Spark to Apache Cassandra connector

• cassandra
• scala
• spark

Base classes to use when writing tests with Spark

MLeap: Deploy ML Pipelines to Production

• scikit-learn
• spark
• data-pipelines
• transformers

Code to accompany Advanced Analytics with Spark from O'Reilly Media

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

• apache-spark
• big-data
• connected-components
• dataframe

ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.

• spark
• big-data
• bioinformatics
• genomics

Project SnappyData - memory optimized analytics database, based on Apache Spark™ and Apache Geode™. Stream, Transact, Analyze, Predict in one cluster

• snappydata
• spark
• memory-database
• analytics

Livy is an open source REST interface for interacting with Apache Spark from anywhere

Sparkling Water provides H2O functionality inside Spark cluster

• h2o
• spark
• machine-learning
• integration

Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.

• livy
• bigdata
• spark
• apachelivy

Essential Spark extensions and helper methods ✨😲

• spark
• dataframe

[DEPRECATED] Tensorflow wrapper for DataFrames on Apache Spark

The Apache Spark - Apache HBase Connector is a library to support Spark accessing HBase table as external data source or sink.

Stream Data Mining Library for Spark Streaming

Apache Spark testing helpers (dependency free & works with Scalatest, uTest, and MUnit)

• spark
• testing-framework

Serverless proxy for Spark cluster

• apache-spark
• api
• big-data
• serverless

Neo4j Connector for Apache Spark, which provides bi-directional read/write access to Neo4j from Spark, using the Spark DataSource APIs

• spark
• neo4j-connector
• cypher
• neo4j-driver