A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.
Repository Index
111,821 visible repositories in the index
1–15 of 15 · Most stars
Fast inference engine for Transformer models
- neural-machine-translation
- cpp
- mkl
- quantization
A repository for storing models that have been inter-converted between various frameworks. Supported frameworks are TensorFlow, PyTorch, ONNX, OpenVINO, TFJS, TFTRT, TensorFlowLite (Float32/16/INT8), EdgeTPU, CoreML.
- tensorflow
- tensorflow-lite
- openvino
- edgetpu
AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.
- quantization
- deep-learning
- compression
- open-source
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
- kv-cache
- llm-inference
- local-llm
- quantization
micronet, a model compression and deploy lib. compression: 1、quantization: quantization-aware-training(QAT), High-Bit(>2b)(DoReFa/Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference)、Low-Bit(≤2b)/Ternary and Binary(TWN/BNN/XNOR-Net); post-training-quantization(PTQ), 8-bit(tensorrt); 2、 pruning: normal、regular and group convolutional channel pruning; 3、 group convolution structure; 4、batch-normalization fuse for quantization. deploy: tensorrt, fp32/fp16/int8(ptq-calibration)、op-adapt(upsample)、dynamic_shape
- quantization
- pruning
- dorefa
- twn
A PyTorch implementation for exploring deep and shallow knowledge distillation (KD) experiments with flexibility
- pytorch
- knowledge-distillation
- deep-neural-networks
- cifar10
Brevitas: neural network quantization in PyTorch
A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning.
Rethinking the Value of Network Pruning (Pytorch) (ICLR 2019)
- convolutional-neural-networks
- network-pruning
- deep-learning
- pytorch
Build, train, optimize, and run computer vision models locally, from raw images to live inference. Open source, optimized for Intel® XPU (CUDA also supported).
- openvino
- computer-vision
- deep-learning
- pytorch
Efficient computing methods developed by Huawei Noah's Ark Lab
- knowledge-distillation
- model-compression
- binary-neural-networks
- pruning
Channel Pruning for Accelerating Very Deep Neural Networks (ICCV'17)
- image-recognition
- model-compression
- acceleration
- object-detection
QKeras: a quantization deep learning library for Tensorflow Keras
- deep-learning
- quantization
- quantized-neural-networks
- hardware-acceleration
Xolotl: run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM): ~33 tok/s on agent tasks, ~41 in fast mode, 512K context. Plus GLM-5.3-Flash, Qwen3.8 and LFM3-Omni. MoE experts tiered across VRAM, pinned RAM and NVMe, with Xolotl Direct NVMe->VRAM. OpenAI + Anthropic API.