Repository Index

LLM inference in C/C++

• ggml

AirLLM 2.8T LLM inference with single 4GB GPU

• chinese-nlp
• finetune
• generative-ai
• instruct-gpt

Tensor library for machine learning

• machine-learning

Enjoy the magic of Diffusion models!

Easy-to-use and powerful LLM and SLM library with awesome model zoo.

• nlp
• embedding
• bert
• ernie

Accessible large language models via k-bit quantization for PyTorch.

• llm
• machine-learning
• pytorch
• qlora

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

• cuda-kernels
• deepspeed
• fastertransformer
• llm-inference

2bit quants, kernels and runtime for consumer devices.

• android
• framework
• ios
• llamacpp

Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.

Fast inference engine for Transformer models

• neural-machine-translation
• cpp
• mkl
• quantization

A fast inference library for running LLMs locally on modern consumer-class GPUs

[ICLR2025, ICML2025, NeurIPS2025 Spotlight] Quantized Attention achieves speedup of 2-5x compared to FlashAttention, without losing end-to-end metrics across language, image, and video models.

• attention
• inference-acceleration
• llm
• quantization

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.

• cuda
• deep-learning
• gpu
• machine-learning

llama.cpp fork with additional SOTA quants and improved performance

4 bits quantization of LLaMA using GPTQ

PyTorch native quantization for training and inference

• brrr
• dtypes
• inference
• mx

An Automatic Model Compression (AutoMC) framework for developing smaller and faster AI applications.

• deep-learning
• model-compression
• mobile-app
• automl

Images to inference with no labeling (use foundation models to train supervised models).

• computer-vision
• model-distillation
• auto-labeling
• deep-learning

An Open-source Toolkit for LLM Development

AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.

• quantization
• deep-learning
• compression
• open-source

SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime

• low-precision
• pruning
• sparsity
• auto-tuning

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

• kv-cache
• llm-inference
• local-llm
• quantization

[ICLR 2020] Contrastive Representation Distillation (CRD), and benchmark of recent knowledge distillation methods

Run Mixtral-8x7B models in Colab or consumer desktops

• colab-notebook
• deep-learning
• google-colab
• language-model