Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Repository Index
87,075 visible repositories in the index
1–24 of 27 · Most stars
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
- bloom
- deep-learning
- distributed-systems
- language-models
Distributed and fault-tolerant realtime computation: stream processing, continuous computation, distributed RPC, and more
Apache Beam is a unified programming model for Batch and Streaming data processing.
- python
- java
- big-data
- beam
Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
- java
- hazelcast
- in-memory
- big-data
A lightweight data processing framework built on DuckDB and 3FS.
- data-processing
- duckdb
Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.
- distributed-computing
- map-reduce
- golang
- distributed-systems
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.
Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.
Mars is a tensor-based unified framework for large-scale data computation which scales numpy, pandas, scikit-learn and Python functions.
A Scala API for Apache Beam and Google Cloud Dataflow.
- scala
- bigquery
- google-cloud
- beam
Run MapReduce jobs on Hadoop or Amazon Web Services
Distributed self-replicating programs in Python
- python
- devops
- infrastructure
- ansible
A cluster computing framework for processing large-scale geospatial data
- cluster-computing
- spatial-sql
- geospatial
- spatial-analysis
Apache DataFusion Ballista Distributed Query Engine
- arrow
- big-data
- dataframe
- olap
Python Stream Processing
- python
- stream-processing
- rust
- data-engineering
a Map/Reduce framework for distributed computing
Distributed Stream Processing
- stream-processing
- stream-processor
- stream-processing-engine
- pony-language
:chart_with_upwards_trend: Adaptive: parallel active learning of mathematical functions
- adaptive-learning
- machine-learning
- python
- live-plots
Cloud-native genomic dataframes and batch computing
- genetics
- vcf
- genomics
- gwas
Essential Spark extensions and helper methods ✨😲
- spark
- dataframe
Phoebus is a distributed framework for large scale graph processing written in Erlang.
Serverless ML Framework