An open, programmable decision layer for models and compute.
- mixture-of-models
- semantic-router
- vllm
- ai-gateway
132,241 visible repositories in the index
1–10 of 10 · Most stars
An open, programmable decision layer for models and compute.
Manages Unified Access to Generative AI Services built on Envoy Gateway
Router that virtually distributes inference across connected devices in the home.
AI Inference Operator for Kubernetes. The easiest way to serve ML models in production. Supports VLMs, LLMs, embeddings, and speech-to-text.
An open-source framework for verifiably private AI inference
LeaderWorkerSet & DisaggregatedSet: Kubernetes APIs for Distributed and Disaggregated AI Serving
🐈 A collection of LLM inference providers and models
Go with your own intelligence - Write Go applications that directly integrate llama.cpp for local inference using hardware acceleration on Linux, macOS, Windows, & WebAssembly.
Berth is a lightweight GPU resource manager written in Go. It controls which applications get access to your GPU, when they get it, and how that access is handed off.
Kubernetes operator for self-hosted LLM inference across a heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. Runtimes: llama.cpp, vLLM, TGI, mlx-server. Multi-GPU sharding, model caching, OpenAI-compatible endpoints. Apache-2.0, run across homelab and on-prem fleets, actively developed.