Repository Index

Apache Spark - A unified analytics engine for large-scale data processing

• python
• scala
• r
• java

Apache Kafka - A distributed event streaming platform

• scala
• kafka
• java
• streaming

Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.

• orchestration
• data-orchestration
• high-availability
• infrastructure-as-code

Apache Flink

• scala
• java
• big-data
• flink

Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.

• database
• distributed-database
• distributed-sql-database
• sql

The most scalable and reliable MQTT broker for AI, IoT, IIoT and connected vehicles

• mqtt
• iot
• mqtt-broker
• erlang

CMAK is a tool for managing Apache Kafka clusters

• kafka
• scala
• cluster-management
• big-data

Scalable real-time messaging server in a language-agnostic way. Self-hosted alternative to Pubnub, Pusher, Ably, socket.io, Phoenix.PubSub, SignalR. Set up once and forever.

• websocket
• redis
• real-time
• messaging

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

• spark
• acid
• big-data
• analytics

The leader in Customer Data Infrastructure

• snowplow
• analytics
• data
• data-pipeline

Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.

• java
• hazelcast
• in-memory
• big-data

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

• dataquality
• spark
• unit-testing
• scala

A distributed MQTT message broker based on Erlang/OTP. Built for high quality & Industrial use cases. The VerneMQ mission is active & the project maintained. Thank you for your support!

• mqtt
• m2m
• scalable
• broker

Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.

• distributed-computing
• map-reduce
• golang
• distributed-systems

In-memory message queue with an Amazon SQS-compatible interface. Runs stand-alone or embedded.

• scala
• sqs-interface
• elasticmq
• amazon-sqs

REST job server for Apache Spark

• spark
• rest-api
• spark-jobserver
• scala

Compositional, streaming I/O library for Scala

• scala
• stream
• streaming-computation
• fp

A cluster computing framework for processing large-scale geospatial data

• cluster-computing
• spatial-sql
• geospatial
• spatial-analysis

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.

• spark
• hive
• sql
• thrift

Apache Spark to Apache Cassandra connector

• cassandra
• scala
• spark

MongooseIM is Erlang Solutions' robust, scalable and efficient XMPP server, aimed at large installations. Specifically designed for enterprise purposes, it is fault-tolerant and can utilise the resources of multiple clustered machines.

• mongooseim
• erlang
• xmpp
• jabber

Probabilistic data structures for processing continuous, unbounded streams.

• bloom-filter
• go
• stable-bloom-filters
• cuckoo-filter

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

• apache-spark
• big-data
• connected-components
• dataframe

Livy is an open source REST interface for interacting with Apache Spark from anywhere