LLM inference in C/C++
- ggml
106,102 visible repositories in the index
1–24 of 238 · Most stars
LLM inference in C/C++
A high-throughput and memory-efficient inference and serving engine for LLMs
Never stop coding. Free MIT AI gateway: one endpoint, 375 providers (150+ free), 1,400+ models Kimi, Claude, GPT, Gemini, GLM, DeepSeek, MiniMax. Works with Claude Code, Codex, Cursor, OpenCode, Cline & Copilot. Quota-aware auto-fallback, RTK+Caveman compression saves 15-95% tokens, MCP/A2A, Desktop/PWA. Built by 900+ contributors
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
Inference code for Llama models
Complete API layer for private AI applications on local models: RAG, skills, tools, MCP, text-to-sql, and more. Works with any OpenAI-compatible inference server.
A hierarchical inference engine for frontier MoE models: hot experts on the GPU (CUDA, Vulkan), warm ones in RAM, cold ones streamed from NVMe. Written in C, runs on the hardware you already own.
Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
AirLLM 2.8T LLM inference with single 4GB GPU
Official inference framework for 1-bit LLMs
SGLang is a high-performance serving framework for large language models and multimodal models.
TranslatedLangchain-Chatchat (formerly langchain-ChatGLM), local knowledge based LLM (like ChatGLM, Qwen and Llama) RAG and Agent app with langchain
Turn any technical book PDF into a Claude Code skill — ready to study, reference, and use while you work.
FastGPT is a knowledge-based platform built on the LLMs, offers a comprehensive suite of out-of-the-box capabilities such as data processing, RAG retrieval, and visual AI workflow orchestration, letting you easily develop and deploy complex question-answering systems without the need for extensive setup or configuration.
Distribute and run LLMs with a single file.
DeepInfer: High-performance, large-scale inference framework for DeepSeek models
DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
Inference Llama 2 in one file of pure C
A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations
FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
Nano vLLM
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.
20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.