Apache Spark - A unified analytics engine for large-scale data processing
- python
- scala
- r
- java
Repository Index
47,133 repositories indexed
1–24 of 32 · Most stars
CMAK is a tool for managing Apache Kafka clusters
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
The leader in Customer Data Infrastructure
- snowplow
- analytics
- data
- data-pipeline
Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
- dataquality
- spark
- unit-testing
- scala
In-memory message queue with an Amazon SQS-compatible interface. Runs stand-alone or embedded.
- scala
- sqs-interface
- elasticmq
- amazon-sqs
REST job server for Apache Spark
Compositional, streaming I/O library for Scala
- scala
- stream
- streaming-computation
- fp
Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.
Apache Spark to Apache Cassandra connector
GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs
- apache-spark
- big-data
- connected-components
- dataframe
Livy is an open source REST interface for interacting with Apache Spark from anywhere
Sparkling Water provides H2O functionality inside Spark cluster
- h2o
- spark
- machine-learning
- integration
Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.
Essential Spark extensions and helper methods ✨😲
- spark
- dataframe
Lightweight real-time big data streaming engine over Apache Pekko
- akka
- stream-processing
- bigdata
- scala
Low-code tool for automating actions on real time data | Stream processing for the users.
- flink
- flink-kafka
- apache-flink
- gui
The Apache Spark - Apache HBase Connector is a library to support Spark accessing HBase table as external data source or sink.
Apache Spark testing helpers (dependency free & works with Scalatest, uTest, and MUnit)
Akka CQRS/ES framework
Serverless proxy for Spark cluster
Neo4j Connector for Apache Spark, which provides bi-directional read/write access to Neo4j from Spark, using the Spark DataSource APIs
- spark
- neo4j-connector
- cypher
- neo4j-driver
A distributed Spark/Scala implementation of the isolation forest and extended isolation forest algorithms for unsupervised outlier detection, featuring support for scalable training and ONNX export for easy cross-platform inference.
- isolation-forest
- spark
- outlier-detection
The LinkedIn Fairness Toolkit (LiFT) is a Scala/Spark library that enables the measurement of fairness in large scale machine learning workflows.
- fairness
- fairness-ml
- fairness-ai