Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Repository Index
71,372 visible repositories in the index
1–24 of 33 · Most stars
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
- bloom
- deep-learning
- distributed-systems
- language-models
Build, Manage and Deploy AI/ML Systems
- machine-learning
- model-management
- ai
- ml
Distributed and fault-tolerant realtime computation: stream processing, continuous computation, distributed RPC, and more
Enterprise job scheduling middleware with distributed computing ability.
- scheduler
- workflow
- distributed
- mapreduce
A unified ecosystem and workspace for large-scale federated learning, bridging the gap between centralized model evolution and decentralized edge intelligence.
High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale
Apache Ignite
- distributed-sql-database
- iot
- osgi
- network-client
A lightweight data processing framework built on DuckDB and 3FS.
- data-processing
- duckdb
Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.
- distributed-computing
- map-reduce
- golang
- distributed-systems
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
The official repository for ROOT: analyzing, storing and visualizing big data, scientifically
- data-analysis
- root-cern
- root
- c-plus-plus
Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.
Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.
Mars is a tensor-based unified framework for large-scale data computation which scales numpy, pandas, scikit-learn and Python functions.
Open MPI main development repository
- c
- hpc
- mpi
- fortran
Run MapReduce jobs on Hadoop or Amazon Web Services
Open-source software for volunteer computing and grid computing.
- boinc
- distributed-computing
- volunteer-computing
- high-performance-computing
Apache Hadoop docker image
- hadoop-docker
- docker-hadoop
- hadoop
- hadoop-cluster
A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.
- spark
- dask
- data-practitioners
- machine-learning
Python Stream Processing
- python
- stream-processing
- rust
- data-engineering
:chart_with_upwards_trend: Adaptive: parallel active learning of mathematical functions
- adaptive-learning
- machine-learning
- python
- live-plots
Cloud-native genomic dataframes and batch computing
- genetics
- vcf
- genomics
- gwas