Repository Index

A high-throughput and memory-efficient inference and serving engine for LLMs

• gpt
• llm
• pytorch
• model-serving

Inference code for Llama models

Complete API layer for private AI applications on local models: RAG, skills, tools, MCP, text-to-sql, and more. Works with any OpenAI-compatible inference server.

• ai
• ai-tools
• on-premise

TranslatedLangchain-Chatchat (formerly Langchain-ChatGLM), a RAG and Agent application based on Langchain and language models such as ChatGLM, Qwen and Llama | Langchain-Chatchat (formerly langchain-ChatGLM), local knowledge based LLM (like ChatGLM, Qwen and Llama) RAG and Agent app with langchain

• chatglm
• langchain
• llm
• knowledge-base

SGLang is a high-performance serving framework for large language models and multimodal models.

• cuda
• inference
• llama
• llm

DeepInfer: High-performance, large-scale inference framework for DeepSeek models

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

• apple-silicon
• inference-server
• llm
• macos

A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations

Nano vLLM

• inference
• llm
• pytorch
• transformer

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

• blackwell
• cuda
• moe
• pytorch

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

• ai
• artificial-intelligence
• deep-learning
• large-language-models

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

• llm
• llmops
• model-inference
• fine-tuning

LMCache: Supercharge Your LLM with the Fastest KV Cache Layer

• amd
• cuda
• inference
• kv-cache

🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

• bloom
• deep-learning
• distributed-systems
• language-models

《深入理解 AI Infra：量化分析与系统设计》（李博杰 著）开源书稿：从硬件约束和模型架构出发，量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验

• accelerator
• ai-infra
• ai-infrastructure
• book

Superduper: End-to-end framework for building custom AI applications and agents.

• ai
• mlops
• torch
• transformers

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

• kv-cache
• llm-inference
• local-llm
• quantization

Open decision models from 0.8B to 27B: a calibrated probability for every option in one pass, and a short thought only when unsure. Engine for vLLM, llama.cpp and MLX.

• agents
• calibration
• classification
• decision-making

Official Inference Code of EAGLE-4

• eagle
• gpu
• llm-inference
• sglang

EdgeRock: Strong Reasoning at Constant Memory. Multimodal models with O(N) generation and O(1) persistent history. Inference and evaluation code.

AI Agent Identity and Access Management and Inference Firewall

• agent-identity
• ai-audit
• ai-firewall
• ai-gateway

Run many AI agents at once on your Mac, up to 3x faster.

• agents
• apple-silicon
• artificial-intelligence
• inference-acceleration

🧱 Experimental library for building inference engines

• inference
• llm
• cuda
• gpt-oss

Our lightning-fast inference engine.