LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
- cuda-kernels
- deepspeed
- fastertransformer
- llm-inference
132,241 visible repositories in the index
1–24 of 26 · Most stars
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.
[MLSys 2024 Best Paper Award] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools
Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
Run Mixtral-8x7B models in Colab or consumer desktops
micronet, a model compression and deploy lib. compression: 1、quantization: quantization-aware-training(QAT), High-Bit(>2b)(DoReFa/Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference)、Low-Bit(≤2b)/Ternary and Binary(TWN/BNN/XNOR-Net); post-training-quantization(PTQ), 8-bit(tensorrt); 2、 pruning: normal、regular and group convolutional channel pruning; 3、 group convolution structure; 4、batch-normalization fuse for quantization. deploy: tensorrt, fp32/fp16/int8(ptq-calibration)、op-adapt(upsample)、dynamic_shape
OneDiff: An out-of-the-box acceleration library for diffusion models.
An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs
[ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
This repository provides code for machine learning algorithms for edge devices developed at Microsoft Research India.
在 oxford hand 数据集上对 YOLOv3 做模型剪枝(network slimming)
A coding-free framework built on PyTorch for reproducible deep learning studies. PyTorch Ecosystem. 🏆28 knowledge distillation methods presented at TPAMI, CVPR, ICLR, ECCV, NeurIPS, ICCV, AAAI, etc are implemented so far. 🎁 Trained models, training logs and configurations are available for ensuring the reproducibiliy and benchmark.
A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning.
Rethinking the Value of Network Pruning (Pytorch) (ICLR 2019)
Pruna is a model optimization framework built for developers, enabling you to deliver faster, more efficient models with minimal overhead.
Run 284B DeepSeek-V4-Flash (<59GiB RAM: 41.1 cold pp, 9.1 N=1 tg, up to 25.3 spec tok/s) & 30B MoEs (22.4 N=1 / 111.6 spec tok/s) on 64GB CPU (0 GPUs).
Build, train, optimize, and run computer vision models locally, from raw images to live inference. Open source, optimized for Intel® XPU (CUDA also supported).
Neural Network Compression Framework for enhanced OpenVINO™ inference
[CVPR 2020] GAN Compression: Efficient Architectures for Interactive Conditional GANs
YOLOv5 Series Multi-backbone(TPH-YOLOv5, Ghostnet, ShuffleNetv2, Mobilenetv3Small, EfficientNetLite, PP-LCNet, SwinTransformer YOLO), Module(CBAM, DCN), Pruning (EagleEye, Network Slimming), Quantization (MQBench) and Deployment (TensorRT, ncnn) Compression Tool Box.
Low-bit LLM inference on CPU/NPU with lookup table
MiniLLM is a minimal system for running modern LLMs on consumer-grade GPUs