Run MoE models bigger than your RAM. Frontier-size MoE on a phone, laptop, edge. Lossless, on stock llama.cpp. An adaptive streaming optimizer. Any hardware, any OS, any model, any quantization, behind a local OpenAI-compatible server.
Low-bit LLM inference on CPU/NPU with lookup table
Deep learning inference nodes for ROS / ROS2 with support for NVIDIA Jetson and TensorRT
jevos: an open-source alternative to Jev and Microsoft-Decision-1. Yes/no decisions, choices and scores with probabilities, on your laptop's CPU.
A scalable inference server for models optimized with OpenVINO™
Bolt is a deep learning library with high performance and heterogeneous flexibility.
A uniform interface to run deep learning models from multiple frameworks
BladeDISC is an end-to-end DynamIc Shape Compiler project for machine learning workloads.
Xbox 360 to PC port of Dante's Inferno using ReXGlue (static recompilation)
Experimental llama.cpp fork for inference research and development
A highly optimized LLM inference acceleration engine for Llama and its variants.
cef util
Suno AI's Bark model in C/C++ for fast text-to-speech generation
Real-time neural network inferencing
Blazing fast Apple Silicon inference engine optimized for Qwen & Nemotron. OpenAI + Anthropic API compatible. Built by folks from nace.ai
oneAPI Data Analytics Library (oneDAL)
Xolotl: run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM): ~33 tok/s on agent tasks, ~41 in fast mode, 512K context. Plus GLM-5.3-Flash, Qwen3.8 and LFM3-Omni. MoE experts tiered across VRAM, pinned RAM and NVMe, with Xolotl Direct NVMe->VRAM. OpenAI + Anthropic API.
llama.cpp fork for significantly improved performance on Ampere generation Nvidia cards (especially RTX 3090 / 3090 Ti): example: 110+ tok/s over a 100K-token generation at temperature 1 for qwen3.8, supporting full 262K ctx. Supports low memory (12gb) configs for 200k+ ctx. 120+ TPS, 3300+ prefill for QFN 3.8.
- ampere
- consumer-gpu
- cuda
- ggml
Inference module for RangeNet++ (milioto2019iros, chen2019iros)
SOTA model-agnostic quantization for GGUF, local inference and beyond. Quantize and run text, image, video and audio models with llama.cpp, vLLM, Diffusers and PyTorch.
- ascend
- audio
- audio-generation
- diffusers
Bayesian Generalized Kernel Inference for Terrain Traversability Mapping
C++/CUDA single-GPU inference engine for Qwen3.8 (incl. Qwen3.8-Flash-Next with offloaded experts) on the RTX 5090; grew from NInfer
small LLM inference engine from scratch, w/ 6+ hours of youtube tutorials & 85+ pages of handwritten notes included
MobiInfer runs 35B-A3B-class MoE models fully on-device on the Qualcomm Hexagon NPU, streaming experts from flash with lookahead scheduling (LES) and DSpark speculation.