Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Repository Index
132,241 visible repositories in the index
1–19 of 19 · Most stars
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
- bloom
- deep-learning
- distributed-systems
- language-models
Build, Manage and Deploy AI/ML Systems
- machine-learning
- model-management
- ai
- ml
Enterprise job scheduling middleware with distributed computing ability.
- scheduler
- workflow
- distributed
- mapreduce
A modular implementation of timely dataflow in Rust
Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.
💎多语言实现,高性能生成唯一数字ID。 💎优化的雪花算法(SnowFlake)——雪花漂移算法,在缩短ID长度的同时,具备极高瞬时并发处理能力(50W/0.1s)。 💎原生支持 C#/Java/Go/Rust/C/JavaScript/TypeScript/Python/Pascal 多语言,提供其它适用于其它语言的多线程安全调用动态库(FFI)。💎支持容器环境自动扩容(自动注册 WorkerId ),单机或分布式唯一IdGenerator。💎顶尖优化,超强效能。
- snowflake
- idgenerator
- dotnet
- rust
Mars is a tensor-based unified framework for large-scale data computation which scales numpy, pandas, scikit-learn and Python functions.
A Scala API for Apache Beam and Google Cloud Dataflow.
- scala
- bigquery
- google-cloud
- beam
Python Stream Processing
- python
- stream-processing
- rust
- data-engineering
Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.
- clickhouse
- simd
- spark-sql
- vectorization
AI based entity resolution, identity resolution, master data management and deduplication at scale for humans and agents
- fuzzymatch
- fuzzy-matching
- deduplication
- dedupe
Baidu Bigflow is an interface that allows for writing distributed computing programs and provides lots of simple, flexible, powerful APIs. Using Bigflow, you can easily handle data of any scale. Bigflow processes 4P+ data inside Baidu and runs about 10k jobs every day.
Apache Celeborn is an elastic and high-performance service for shuffle and spilled data.
Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.
- hpc
- workflow
- workflow-automation
- workflow-management
Map-Reduce for Clojure
Distributed Temporal Graph Analytics with Apache Flink
- graph
- property-graph
- distributed-graph-analytics
- apache-flink
Distributed batch data processing framework
Parallel Workflow extension for Rake, runs on multicores, clusters, clouds.
- ruby
- rake
- pwrake
- parallel