🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
- bloom
- deep-learning
- distributed-systems
- language-models
142,273 visible repositories in the index
1–23 of 23 · Most stars
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
Build, Manage and Deploy AI/ML Systems
Distributed and fault-tolerant realtime computation: stream processing, continuous computation, distributed RPC, and more
Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
Data-Centric Pipelines and Data Versioning
High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale
Apache Ignite
Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.
Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.
Open-source software for volunteer computing and grid computing.
YTsaurus is a scalable and fault-tolerant open-source big data platform.
Feathr – A scalable, unified data and AI engineering platform for enterprise
Python 3.8+ toolbox for submitting jobs to Slurm
AI based entity resolution, identity resolution, master data management and deduplication at scale for humans and agents
Cloud-native genomic dataframes and batch computing
Apache Celeborn is an elastic and high-performance service for shuffle and spilled data.
High-performance, high-scalability distributed computing for the BEAM.
Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.
The repository host gives no description.
Open Lakehouse Compute Fabric: an open-source lakehouse platform with Microsoft Fabric Compute Extension.
Distribute work across machines using the lparallel API.
Parallel Workflow extension for Rake, runs on multicores, clusters, clouds.
The core libraries of the teknek stream processing platform