Kubernetes operator for self-hosted LLM inference across a heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. Runtimes: llama.cpp, vLLM, TGI, mlx-server. Multi-GPU sharding, model caching, OpenAI-compatible endpoints. Apache-2.0, run across homelab and on-prem fleets, actively developed.
localaik is a local compatibility server for a subset of the Gemini and OpenAI APIs. Run one container locally, point your SDK to it, and the proxy serves both protocol shapes on the same port for tests and development.