Repository Index

Apache Spark - A unified analytics engine for large-scale data processing

• python
• scala
• r
• java

Distributed and fault-tolerant realtime computation: stream processing, continuous computation, distributed RPC, and more

Apache Beam is a unified programming model for Batch and Streaming data processing.

• python
• java
• big-data
• beam

Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.

• java
• hazelcast
• in-memory
• big-data

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

• machine-learning
• python
• data-engineering
• distributed-computing

A lightweight data processing framework built on DuckDB and 3FS.

• data-processing
• duckdb

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

• arrow
• big-data
• pyspark
• rust

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

A cluster computing framework for processing large-scale geospatial data

• cluster-computing
• spatial-sql
• geospatial
• spatial-analysis

Python Stream Processing

• python
• stream-processing
• rust
• data-engineering

Distributed Stream Processing

• stream-processing
• stream-processor
• stream-processing-engine
• pony-language

Phoebus is a distributed framework for large scale graph processing written in Erlang.

Distributed batch data processing framework

• cassandra
• rabbitmq
• go
• batch-processing

The core libraries of the teknek stream processing platform