Apache Spark - A unified analytics engine for large-scale data processing
- python
- scala
- r
- java
Repository Index
85,081 visible repositories in the index
1–24 of 29 · Most stars
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Distributed and fault-tolerant realtime computation: stream processing, continuous computation, distributed RPC, and more
Apache Beam is a unified programming model for Batch and Streaming data processing.
- python
- java
- big-data
- beam
Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
- java
- hazelcast
- in-memory
- big-data
Data-Centric Pipelines and Data Versioning
A lightweight data processing framework built on DuckDB and 3FS.
- data-processing
- duckdb
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
The official repository for ROOT: analyzing, storing and visualizing big data, scientifically
- data-analysis
- root-cern
- root
- c-plus-plus
Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.
Open MPI main development repository
- c
- hpc
- mpi
- fortran
Distributed self-replicating programs in Python
- python
- devops
- infrastructure
- ansible
A cluster computing framework for processing large-scale geospatial data
- cluster-computing
- spatial-sql
- geospatial
- spatial-analysis
Apache DataFusion Ballista Distributed Query Engine
- arrow
- big-data
- dataframe
- olap
Python Stream Processing
- python
- stream-processing
- rust
- data-engineering
Apache Auron is an accelerator for big data engines, leveraging native vectorized execution to accelerate query processing.
Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.
- clickhouse
- simd
- spark-sql
- vectorization
Distributed Stream Processing
- stream-processing
- stream-processor
- stream-processing-engine
- pony-language
Essential Spark extensions and helper methods ✨😲
- spark
- dataframe
Phoebus is a distributed framework for large scale graph processing written in Erlang.
Open Lakehouse Compute Fabric: an open-source lakehouse platform with Microsoft Fabric Compute Extension.
- apache-iceberg
- apache-spark
- data-engineering
- delta-lake
Python MapReduce library written in Cython. Visit us in #hadoopy on freenode. See the link below for documentation and tutorials.
Distribute work across machines using the lparallel API.