Run frontier AI locally.
Repository Index
111,820 visible repositories in the index
1–24 of 26 · Most stars
Apache Spark - A unified analytics engine for large-scale data processing
- python
- scala
- r
- java
Distributed and fault-tolerant realtime computation: stream processing, continuous computation, distributed RPC, and more
Apache Beam is a unified programming model for Batch and Streaming data processing.
- python
- java
- big-data
- beam
Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
- java
- hazelcast
- in-memory
- big-data
Data-Centric Pipelines and Data Versioning
High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale
A lightweight data processing framework built on DuckDB and 3FS.
- data-processing
- duckdb
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
A cluster computing framework for processing large-scale geospatial data
- cluster-computing
- spatial-sql
- geospatial
- spatial-analysis
Apache Hadoop docker image
- hadoop-docker
- docker-hadoop
- hadoop
- hadoop-cluster
Python Stream Processing
- python
- stream-processing
- rust
- data-engineering
Apache Spark docker image
- spark-kubernetes
- kubernetes
- k8s-spark
- docker
Apache Auron is an accelerator for big data engines, leveraging native vectorized execution to accelerate query processing.
Run Hadoop Custer within Docker Containers
Distributed Stream Processing
- stream-processing
- stream-processor
- stream-processing-engine
- pony-language
Serverless batch computing made simple.
- typescript
- javascript
- serverless
- batch
Data processing on Hadoop without the hassle.
Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.
C implementation of the Raft Consensus protocol, BSD licensed
- raft-consensus-algorithm
Baidu Bigflow is an interface that allows for writing distributed computing programs and provides lots of simple, flexible, powerful APIs. Using Bigflow, you can easily handle data of any scale. Bigflow processes 4P+ data inside Baidu and runs about 10k jobs every day.
Phoebus is a distributed framework for large scale graph processing written in Erlang.
Python MapReduce library written in Cython. Visit us in #hadoopy on freenode. See the link below for documentation and tutorials.