LLM inference in C/C++
- ggml
Repository Index
47,133 repositories indexed
1–24 of 63 · Most stars
A high-throughput and memory-efficient inference and serving engine for LLMs
Never stop coding. Free MIT AI gateway: one endpoint, 375 providers (150+ free), 1,400+ models Kimi, Claude, GPT, Gemini, GLM, DeepSeek, MiniMax. Works with Claude Code, Codex, Cursor, OpenCode, Cline & Copilot. Quota-aware auto-fallback, RTK+Caveman compression saves 15-95% tokens, MCP/A2A, Desktop/PWA. Built by 900+ contributors
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
- llm-inference
- ai-chat
Inference code for Llama models
Complete API layer for private AI applications on local models: RAG, skills, tools, MCP, text-to-sql, and more. Works with any OpenAI-compatible inference server.
AirLLM 2.8T LLM inference with single 4GB GPU
- chinese-nlp
- finetune
- generative-ai
- instruct-gpt
Official inference framework for 1-bit LLMs
Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
TranslatedLangchain-Chatchat (formerly langchain-ChatGLM), local knowledge based LLM (like ChatGLM, Qwen and Llama) RAG and Agent app with langchain
- chatglm
- langchain
- llm
- knowledge-base
SGLang is a high-performance serving framework for large language models and multimodal models.
Distribute and run LLMs with a single file.
- cross-platform
- gguf
- llama-cpp
- local-ai
DeepInfer: High-performance, large-scale inference framework for DeepSeek models
DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
Inference Llama 2 in one file of pure C
A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations
FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
- deepseek-v4
- edge
- glm
- inference
Nano vLLM
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.
20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
- llm
- llmops
- model-inference
- fine-tuning
A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.
- avx2
- c99
- cpu-inference
- deep-learning
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
- bloom
- deep-learning
- distributed-systems
- language-models