Apache Spark - A unified analytics engine for large-scale data processing
- python
- scala
- r
- java
100,822 visible repositories in the index
1–24 of 39 · Most stars
Apache Spark - A unified analytics engine for large-scale data processing
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
Build, Manage and Deploy AI/ML Systems
Apache Beam is a unified programming model for Batch and Streaming data processing.
Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale
Apache Ignite
Distributed AI/LLM for the people. Share compute privately or publicly to power your agents and chat.
Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
The official repository for ROOT: analyzing, storing and visualizing big data, scientifically
Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.
Mars is a tensor-based unified framework for large-scale data computation which scales numpy, pandas, scikit-learn and Python functions.
Open MPI main development repository
A Scala API for Apache Beam and Google Cloud Dataflow.
Open-source software for volunteer computing and grid computing.
A cluster computing framework for processing large-scale geospatial data
YTsaurus is a scalable and fault-tolerant open-source big data platform.
A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.
Feathr – A scalable, unified data and AI engineering platform for enterprise
Apache Auron is an accelerator for big data engines, leveraging native vectorized execution to accelerate query processing.
A distributed task scheduler for Dask
Python 3.8+ toolbox for submitting jobs to Slurm