Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Repository Index
109,091 visible repositories in the index
1–24 of 28 · Most stars
Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.
Enterprise job scheduling middleware with distributed computing ability.
- scheduler
- workflow
- distributed
- mapreduce
Apache Ignite
- distributed-sql-database
- iot
- osgi
- network-client
A lightweight data processing framework built on DuckDB and 3FS.
- data-processing
- duckdb
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.
Mars is a tensor-based unified framework for large-scale data computation which scales numpy, pandas, scikit-learn and Python functions.
Run MapReduce jobs on Hadoop or Amazon Web Services
Distributed self-replicating programs in Python
- python
- devops
- infrastructure
- ansible
A cluster computing framework for processing large-scale geospatial data
- cluster-computing
- spatial-sql
- geospatial
- spatial-analysis
nebulai's depin computing services
YTsaurus is a scalable and fault-tolerant open-source big data platform.
- big-data
- clickhouse
- distributed-database
- lakehouse
A new arguably faster implementation of Apache Spark from scratch in Rust
rust raft with improvements
- consensus
- distributed-systems
- raft
Feathr – A scalable, unified data and AI engineering platform for enterprise
- feature-engineering
- feature-store
- artificial-intelligence
- mlops
Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.
- clickhouse
- simd
- spark-sql
- vectorization
Serverless batch computing made simple.
- typescript
- javascript
- serverless
- batch
AI based entity resolution, identity resolution, master data management and deduplication at scale for humans and agents
- fuzzymatch
- fuzzy-matching
- deduplication
- dedupe
Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.
Making data lake work for time series
- data-lake-analytics
- distributed
- etl-framework
- mlops
C implementation of the Raft Consensus protocol, BSD licensed
- raft-consensus-algorithm
Baidu Bigflow is an interface that allows for writing distributed computing programs and provides lots of simple, flexible, powerful APIs. Using Bigflow, you can easily handle data of any scale. Bigflow processes 4P+ data inside Baidu and runs about 10k jobs every day.