Apache Spark - A unified analytics engine for large-scale data processing
- python
- scala
- r
- java
95,842 visible repositories in the index
1–24 of 30 · Most stars
Apache Spark - A unified analytics engine for large-scale data processing
Data-Centric Pipelines and Data Versioning
High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale
Distributed AI/LLM for the people. Share compute privately or publicly to power your agents and chat.
Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.
🔨 🍇 💻 🚀 GraphScope: A One-Stop Large-Scale Graph Computing System from Alibaba | 一站式图计算系统
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
The official repository for ROOT: analyzing, storing and visualizing big data, scientifically
Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.
Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.
Mars is a tensor-based unified framework for large-scale data computation which scales numpy, pandas, scikit-learn and Python functions.
Run MapReduce jobs on Hadoop or Amazon Web Services
Distributed self-replicating programs in Python
Open-source software for volunteer computing and grid computing.
nebulai's depin computing services
YTsaurus is a scalable and fault-tolerant open-source big data platform.
Feathr – A scalable, unified data and AI engineering platform for enterprise
Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.
:chart_with_upwards_trend: Adaptive: parallel active learning of mathematical functions
Cloud-native genomic dataframes and batch computing
Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.
Haskell on Apache Spark.
Phoebus is a distributed framework for large scale graph processing written in Erlang.