A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.
Repository Index
99,825 visible repositories in the index
1–13 of 13 · Most stars
🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools
Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.
- knowledge-distillation
- model-compression
- quantization
- pretrained-models
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
- kv-cache
- llm-inference
- local-llm
- quantization
Olive: Simplify ML Model Finetuning, Conversion, Quantization, and Optimization for CPUs, GPUs and NPUs.
OneDiff: An out-of-the-box acceleration library for diffusion models.
- comfyui
- diffusers
- pytorch
- sdxl
An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs
TurboQuant: Near-optimal KV cache quantization for LLM inference (3-bit keys, 2-bit values) with Triton kernels + vLLM integration
A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning.
Pruna is a model optimization framework built for developers, enabling you to deliver faster, more efficient models with minimal overhead.
Build, train, optimize, and run computer vision models locally, from raw images to live inference. Open source, optimized for Intel® XPU (CUDA also supported).
- openvino
- computer-vision
- deep-learning
- pytorch
LLM model quantization (compression) toolkit with HW acceleration support for Nvidia CUDA, AMD ROCm, Intel XPU, Tenstorrent, Huawei Ascent NPU, and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.
- gptq
- optimum
- peft
- quantization
PrismQuant: Optimal Null-Space Rotations for Grouped Quantizers
- int4
- large-language-models
- pytorch
- quantization