Repository Index

LLM inference in C/C++

• ggml

Tensor library for machine learning

• machine-learning

Accessible large language models via k-bit quantization for PyTorch.

• llm
• machine-learning
• pytorch
• qlora

A fast inference library for running LLMs locally on modern consumer-class GPUs

Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.

• knowledge-distillation
• model-compression
• quantization
• pretrained-models

SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime

• low-precision
• pruning
• sparsity
• auto-tuning

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

• kv-cache
• llm-inference
• local-llm
• quantization

An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs

[ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

a systemone proxy using embeddings that distills a teacher model

PrismQuant: Optimal Null-Space Rotations for Grouped Quantizers

• int4
• large-language-models
• pytorch
• quantization