Easy-to-use and powerful LLM and SLM library with awesome model zoo.
- nlp
- embedding
- bert
- ernie
132,183 visible repositories in the index
1–24 of 53 · Most stars
Easy-to-use and powerful LLM and SLM library with awesome model zoo.
A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.
The central nervous system and backend orchestrator for Fractal. It handles multi-tenant task scheduling, real-time TFLOPs budgeting, and federated weight aggregation using FedAvg.
Fast inference engine for Transformer models
A fast inference library for running LLMs locally on modern consumer-class GPUs
llama.cpp fork with additional SOTA quants and improved performance
[CVPR 2023] DepGraph: Towards Any Structural Pruning; LLMs, Vision Foundation Models, etc.
Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.
An Automatic Model Compression (AutoMC) framework for developing smaller and faster AI applications.
Images to inference with no labeling (use foundation models to train supervised models).
An Open-source Toolkit for LLM Development
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.
SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime
Base pretrained models and datasets in pytorch (MNIST, SVHN, CIFAR10, CIFAR100, STL10, AlexNet, VGG16, VGG19, ResNet, Inception, SqueezeNet)
Run Mixtral-8x7B models in Colab or consumer desktops
micronet, a model compression and deploy lib. compression: 1、quantization: quantization-aware-training(QAT), High-Bit(>2b)(DoReFa/Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference)、Low-Bit(≤2b)/Ternary and Binary(TWN/BNN/XNOR-Net); post-training-quantization(PTQ), 8-bit(tensorrt); 2、 pruning: normal、regular and group convolutional channel pruning; 3、 group convolution structure; 4、batch-normalization fuse for quantization. deploy: tensorrt, fp32/fp16/int8(ptq-calibration)、op-adapt(upsample)、dynamic_shape
A PyTorch implementation for exploring deep and shallow knowledge distillation (KD) experiments with flexibility
An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs
PPL Quantization Tool (PPQ) is a powerful offline neural network quantization tool.
TurboQuant: Near-optimal KV cache quantization for LLM inference (3-bit keys, 2-bit values) with Triton kernels + vLLM integration
A PyTorch-based knowledge distillation toolkit for natural language processing
This repository provides code for machine learning algorithms for edge devices developed at Microsoft Research India.
Efficient, scalable and enterprise-grade CPU/GPU inference server for 🤗 Hugging Face transformer models 🚀