An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs
Repository Index
111,821 visible repositories in the index
97–120 of 193 · Most stars
Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs
A native macOS AI chat app powered by MLX. 100% local inference on Apple Silicon, no cloud required. Built with ShipSwift.
- app
- klee
- self-hosted
- apple-silicon
Router that virtually distributes inference across connected devices in the home.
Open decision models from 0.8B to 27B: a calibrated probability for every option in one pass, and a short thought only when unsure. Engine for vLLM, llama.cpp and MLX.
- agents
- calibration
- classification
- decision-making
Pure C inference of Mistral Voxtral Realtime 4B speech to text model
Route, manage, and analyze your LLM requests across multiple providers with a unified API interface.
NobodyWho is an inference engine that lets you run LLMs locally and efficiently on any device.
High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.
A high-performance inference engine for LLM, VLM and DiT models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.
- deepseek
- inference
- inference-engine
- llm-inference
The open LLM inference engine for AWS Trainium: NKI megakernels, 1M-token context, OpenAI-compatible API. By RunAnywhere.
INT4/INT5/INT8 and FP16 inference on CPU for RWKV language model
- deep-learning
- language-model
- llm
- machine-learning
V100 / SM70-focused vLLM engineering fork for modern LLM inference.
Jev-compatible API server for open decision models. Ollama for decision models: pull and serve 29 model families (Laya, pplx-decider, Clef, Winnow, coba...) locally or self-hosted behind /v1/systemone. The TypeSafe SDK works unchanged: set its base URL and pick an open model. CLI, desktop app, Docker, MCP server, public bench.
- decision-models
- jev
- laya
- local-inference
Parallax is a distributed model serving framework that lets you build your own AI cluster anywhere
RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.
Local inference that behaves like a frontier API: Qwen3.8-Flash-Next on AMD Strix Halo plus small NPU models, over the OpenAI and Anthropic APIs on one port.
- amd
- gfx1151
- gpu-inference
- hip
KVMem virtualizes KV memory across inference backends to make ultra-long-context local AI accessible on limited hardware.
Run 284B DeepSeek-V4-Flash (<59GiB RAM: 41.1 cold pp, 9.1 N=1 tg, up to 25.3 spec tok/s) & 30B MoEs (22.4 N=1 / 111.6 spec tok/s) on 64GB CPU (0 GPUs).
Highly Performant, Modular, Memory Safe and Production-ready Inference, Ingestion and Indexing built in Rust 🦀
- machine-learning
- indexing
- rag
- rust
OpenTelemetry Instrumentation for AI Observability
Offline inference engine for art, real-time voice conversations, LLM powered chatbots and automated workflows
- ai
- ai-art
- art
- asset-generator
Jlama is a modern LLM inference engine for Java
AI Inference Operator for Kubernetes. The easiest way to serve ML models in production. Supports VLMs, LLMs, embeddings, and speech-to-text.
- k8s
- llm
- openai-api
- autoscaler