A Datacenter Scale Distributed Inference Serving Framework
- diffusion
- disaggregated-serving
- kubernetes
- llm-inference
132,241 visible repositories in the index
1–24 of 47 · Most stars
A Datacenter Scale Distributed Inference Serving Framework
Open source inference engine that tunes its kernels on your hardware to find the fastest way to run each model. 2x faster than Ollama. Works on Apple Silicon, NVIDIA, AMD, Intel, or just a CPU.
Fast, flexible LLM inference
Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic.
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
A high-performance LLM inference API and Chat UI that integrates DeepSeek R1's CoT reasoning traces with Anthropic Claude models.
A blazing fast inference solution for text embeddings models
RuVector provides High Performance, Real-Time decisions and agent memory , Self-Learning Ai, Vector GNN DB built in Rust.
TypeDB: Built for systems, not records
Building a modern functional compiler from first principles.
A static, type inferred and embeddable language written in Rust.
A high-performance inference engine for AI models
Framework to prove inference of ML models blazingly fast
Add a real-time analytics node to your operational database. Spice is a portable, accelerated SQL query, search, and LLM-inference engine in Rust for data-grounded AI apps and agents.
Distributed inference for mobile, desktop and server.
Tiny, no-nonsense, self-contained, Tensorflow and ONNX inference
OpenLake is a high performance storage engine for efficient LLM inference and GPU Training
Inference at the speed of light.
Fast ML inference & training for ONNX models in Rust
A Rust port of FlameGraph
AICI: Prompts as (Wasm) Programs
Unified, efficient, and easy-to-use inference infrastructure for edge AI || 面向端侧 AI 的统一、高效、易用的推理基础设施
NobodyWho is an inference engine that lets you run LLMs locally and efficiently on any device.
Import .sql files into TypeScript with full type inference and zero codegen.