Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Repository Index
132,241 visible repositories in the index
1–17 of 17 · Most stars
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
- bloom
- deep-learning
- distributed-systems
- language-models
Build, Manage and Deploy AI/ML Systems
- machine-learning
- model-management
- ai
- ml
Apache Beam is a unified programming model for Batch and Streaming data processing.
- python
- java
- big-data
- beam
Enterprise job scheduling middleware with distributed computing ability.
- scheduler
- workflow
- distributed
- mapreduce
Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.
A Scala API for Apache Beam and Google Cloud Dataflow.
- scala
- bigquery
- google-cloud
- beam
Distributed self-replicating programs in Python
- python
- devops
- infrastructure
- ansible
Python Stream Processing
- python
- stream-processing
- rust
- data-engineering
Python 3.8+ toolbox for submitting jobs to Slurm
- slurm
- python
- clusters
Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.
- clickhouse
- simd
- spark-sql
- vectorization
Cloud-native genomic dataframes and batch computing
- genetics
- vcf
- genomics
- gwas
:rocket: R package: future: Unified Parallel and Distributed Processing in R for Everyone
- r
- cran
- parallel-processing
- parallel-computing
Open Lakehouse Compute Fabric: an open-source lakehouse platform with Microsoft Fabric Compute Extension.
- apache-iceberg
- apache-spark
- data-engineering
- delta-lake
Distributed Temporal Graph Analytics with Apache Flink
- graph
- property-graph
- distributed-graph-analytics
- apache-flink