LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
- cuda-kernels
- deepspeed
- fastertransformer
- llm-inference
89,572 visible repositories in the index
1–20 of 20 · Most stars
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools
Images to inference with no labeling (use foundation models to train supervised models).
An Open-source Toolkit for LLM Development
Olive: Simplify ML Model Finetuning, Conversion, Quantization, and Optimization for CPUs, GPUs and NPUs.
Run Mixtral-8x7B models in Colab or consumer desktops
micronet, a model compression and deploy lib. compression: 1、quantization: quantization-aware-training(QAT), High-Bit(>2b)(DoReFa/Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference)、Low-Bit(≤2b)/Ternary and Binary(TWN/BNN/XNOR-Net); post-training-quantization(PTQ), 8-bit(tensorrt); 2、 pruning: normal、regular and group convolutional channel pruning; 3、 group convolution structure; 4、batch-normalization fuse for quantization. deploy: tensorrt, fp32/fp16/int8(ptq-calibration)、op-adapt(upsample)、dynamic_shape
PPL Quantization Tool (PPQ) is a powerful offline neural network quantization tool.
Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.
A PyTorch-based knowledge distillation toolkit for natural language processing
OpenMMLab Model Compression Toolbox and Benchmark.
在 oxford hand 数据集上对 YOLOv3 做模型剪枝(network slimming)
A coding-free framework built on PyTorch for reproducible deep learning studies. PyTorch Ecosystem. 🏆28 knowledge distillation methods presented at TPAMI, CVPR, ICLR, ECCV, NeurIPS, ICCV, AAAI, etc are implemented so far. 🎁 Trained models, training logs and configurations are available for ensuring the reproducibiliy and benchmark.
PaddleSlim is an open-source library for deep model compression and architecture search.
A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning.
yolov3 yolov4 channel and layer pruning, Knowledge Distillation 层剪枝,通道剪枝,知识蒸馏
Run 284B DeepSeek-V4-Flash (<59GiB RAM: 41.1 cold pp, 9.1 N=1 tg, up to 25.3 spec tok/s) & 30B MoEs (22.4 N=1 / 111.6 spec tok/s) on 64GB CPU (0 GPUs).
OmniEdu: Open Foundation Models for Learning and Teaching
Xolotl: run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM): ~33 tok/s on agent tasks, ~41 in fast mode, 512K context. Plus GLM-5.3-Flash, Qwen3.8 and LFM3-Omni. MoE experts tiered across VRAM, pinned RAM and NVMe, with Xolotl Direct NVMe->VRAM. OpenAI + Anthropic API.
[Tech Report] Simplifying and Improving On-Policy Distillation for Few-step Diffusion