Apache Spark - A unified analytics engine for large-scale data processing
Repository Index
47,133 repositories indexed
1–24 of 24 · Most stars
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark
State of the Art Natural Language Processing
Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
REST job server for Apache Spark
Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.
Apache Spark to Apache Cassandra connector
MLeap: Deploy ML Pipelines to Production
GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs
ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.
Project SnappyData - memory optimized analytics database, based on Apache Spark™ and Apache Geode™. Stream, Transact, Analyze, Predict in one cluster
Sparkling Water provides H2O functionality inside Spark cluster
Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.
Essential Spark extensions and helper methods ✨😲
Apache Spark testing helpers (dependency free & works with Scalatest, uTest, and MUnit)
Neo4j Connector for Apache Spark, which provides bi-directional read/write access to Neo4j from Spark, using the Spark DataSource APIs
A distributed Spark/Scala implementation of the isolation forest and extended isolation forest algorithms for unsupervised outlier detection, featuring support for scalable training and ONNX export for easy cross-platform inference.
The LinkedIn Fairness Toolkit (LiFT) is a Scala/Spark library that enables the measurement of fairness in large scale machine learning workflows.
An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.
The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
A library that brings useful functions from various modern database management systems to Apache Spark
An open-source toolkit for analyzing line-oriented JSON Twitter archives with Apache Spark.
Spark plugin to retrieve metrics from a variety of cluster resources