Repository Index

AirLLM 2.8T LLM inference with single 4GB GPU

• chinese-nlp
• finetune
• generative-ai
• instruct-gpt

Easy-to-use and powerful LLM and SLM library with awesome model zoo.

• nlp
• embedding
• bert
• ernie

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

• cuda-kernels
• deepspeed
• fastertransformer
• llm-inference

The repository host gives no description.

A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.

Fast inference engine for Transformer models

• neural-machine-translation
• cpp
• mkl
• quantization

A fast inference library for running LLMs locally on modern consumer-class GPUs

[ICLR2025, ICML2025, NeurIPS2025 Spotlight] Quantized Attention achieves speedup of 2-5x compared to FlashAttention, without losing end-to-end metrics across language, image, and video models.

• attention
• inference-acceleration
• llm
• quantization

TurboDiffusion: 100–200× Acceleration for Video Diffusion Models

• ai-infra
• diffusion-models
• inference-acceleration
• rcm

[CVPR 2023] DepGraph: Towards Any Structural Pruning; LLMs, Vision Foundation Models, etc.

• pruning
• model-compression
• efficient-deep-learning
• llm

An Open-source Toolkit for LLM Development

AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.

• quantization
• deep-learning
• compression
• open-source

SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime

• low-precision
• pruning
• sparsity
• auto-tuning

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

• kv-cache
• llm-inference
• local-llm
• quantization

Code for the ICLR 2023 paper "GPTQ: Accurate Post-training Quantization of Generative Pretrained Transformers".

Run Mixtral-8x7B models in Colab or consumer desktops

• colab-notebook
• deep-learning
• google-colab
• language-model

micronet, a model compression and deploy lib. compression: 1、quantization: quantization-aware-training(QAT), High-Bit(>2b)(DoReFa/Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference)、Low-Bit(≤2b)/Ternary and Binary(TWN/BNN/XNOR-Net); post-training-quantization(PTQ), 8-bit(tensorrt); 2、 pruning: normal、regular and group convolutional channel pruning; 3、 group convolution structure; 4、batch-normalization fuse for quantization. deploy: tensorrt, fp32/fp16/int8(ptq-calibration)、op-adapt(upsample)、dynamic_shape

• quantization
• pruning
• dorefa
• twn

A PyTorch implementation for exploring deep and shallow knowledge distillation (KD) experiments with flexibility

• pytorch
• knowledge-distillation
• deep-neural-networks
• cifar10

OneDiff: An out-of-the-box acceleration library for diffusion models.

• comfyui
• diffusers
• pytorch
• sdxl

PPL Quantization Tool (PPQ) is a powerful offline neural network quantization tool.

• neural-network
• deep-learning
• quantization
• pytorch

An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs

TurboQuant: Near-optimal KV cache quantization for LLM inference (3-bit keys, 2-bit values) with Triton kernels + vLLM integration

A PyTorch-based knowledge distillation toolkit for natural language processing

• bert
• pytorch
• nlp
• knowledge

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率，量化感知训练的精度

• int4
• quantization
• rounding
• transformers