Repository Index

Apache Spark - A unified analytics engine for large-scale data processing

• python
• scala
• r
• java

CMAK is a tool for managing Apache Kafka clusters

• kafka
• scala
• cluster-management
• big-data

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

• spark
• acid
• big-data
• analytics

The leader in Customer Data Infrastructure

• snowplow
• analytics
• data
• data-pipeline

Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark

• spark
• pyspark
• azure
• scala

State of the Art Natural Language Processing

• nlp
• natural-language-processing
• spark
• pyspark

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

• dataquality
• spark
• unit-testing
• scala

In-memory message queue with an Amazon SQS-compatible interface. Runs stand-alone or embedded.

• scala
• sqs-interface
• elasticmq
• amazon-sqs

REST job server for Apache Spark

• spark
• rest-api
• spark-jobserver
• scala

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

• spark
• hive
• sql
• thrift

Apache Spark to Apache Cassandra connector

• cassandra
• scala
• spark

MLeap: Deploy ML Pipelines to Production

• scikit-learn
• spark
• data-pipelines
• transformers

Distributed Prometheus time series database

• scala
• time-series
• database
• distributed

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

• apache-spark
• big-data
• connected-components
• dataframe

ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.

• spark
• big-data
• bioinformatics
• genomics

Project SnappyData - memory optimized analytics database, based on Apache Spark™ and Apache Geode™. Stream, Transact, Analyze, Predict in one cluster

• snappydata
• spark
• memory-database
• analytics

Livy is an open source REST interface for interacting with Apache Spark from anywhere

Sparkling Water provides H2O functionality inside Spark cluster

• h2o
• spark
• machine-learning
• integration

Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.

• livy
• bigdata
• spark
• apachelivy

Essential Spark extensions and helper methods ✨😲

• spark
• dataframe

Lightweight real-time big data streaming engine over Apache Pekko

• akka
• stream-processing
• bigdata
• scala

Low-code tool for automating actions on real time data | Stream processing for the users.

• flink
• flink-kafka
• apache-flink
• gui

[DEPRECATED] Tensorflow wrapper for DataFrames on Apache Spark

The Apache Spark - Apache HBase Connector is a library to support Spark accessing HBase table as external data source or sink.