A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.
Repository Index
97,836 visible repositories in the index
1–10 of 10 · Most stars
🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools
Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.
- knowledge-distillation
- model-compression
- quantization
- pretrained-models
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
- kv-cache
- llm-inference
- local-llm
- quantization
Olive: Simplify ML Model Finetuning, Conversion, Quantization, and Optimization for CPUs, GPUs and NPUs.
OneDiff: An out-of-the-box acceleration library for diffusion models.
- comfyui
- diffusers
- pytorch
- sdxl
An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs
A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning.
Pruna is a model optimization framework built for developers, enabling you to deliver faster, more efficient models with minimal overhead.
Build, train, optimize, and run computer vision models locally, from raw images to live inference. Open source, optimized for Intel® XPU (CUDA also supported).
- openvino
- computer-vision
- deep-learning
- pytorch