High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale
Repository Index
132,241 visible repositories in the index
1–13 of 13 · Most stars
Distributed AI/LLM for the people. Share compute privately or publicly to power your agents and chat.
- agents
- ai
- decentralized
- distributed
A modular implementation of timely dataflow in Rust
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
💎多语言实现,高性能生成唯一数字ID。 💎优化的雪花算法(SnowFlake)——雪花漂移算法,在缩短ID长度的同时,具备极高瞬时并发处理能力(50W/0.1s)。 💎原生支持 C#/Java/Go/Rust/C/JavaScript/TypeScript/Python/Pascal 多语言,提供其它适用于其它语言的多线程安全调用动态库(FFI)。💎支持容器环境自动扩容(自动注册 WorkerId ),单机或分布式唯一IdGenerator。💎顶尖优化,超强效能。
- snowflake
- idgenerator
- dotnet
- rust
YTsaurus is a scalable and fault-tolerant open-source big data platform.
- big-data
- clickhouse
- distributed-database
- lakehouse
A new arguably faster implementation of Apache Spark from scratch in Rust
Apache DataFusion Ballista Distributed Query Engine
- arrow
- big-data
- dataframe
- olap
rust raft with improvements
- consensus
- distributed-systems
- raft
Python Stream Processing
- python
- stream-processing
- rust
- data-engineering
Apache Auron is an accelerator for big data engines, leveraging native vectorized execution to accelerate query processing.
Apache DataFusion Comet Spark Accelerator
- arrow
- datafusion
- rust
- spark
Serverless ML Framework