Accessible large language models via k-bit quantization for PyTorch.
- llm
- machine-learning
- pytorch
- qlora
110,088 visible repositories in the index
1–10 of 10 · Most stars
Accessible large language models via k-bit quantization for PyTorch.
Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime
[ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
Spiking Brain-inspired Large Models, integrating hybrid efficient attention, MoE modules and spike encoding into its architecture
Official Pytorch repository for Extreme Compression of Large Language Models via Additive Quantization https://arxiv.org/pdf/2401.06118.pdf and PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression https://arxiv.org/abs/2405.14852
LLM KV cache compression made easy
[NeurIPS 2023] LLM-Pruner: On the Structural Pruning of Large Language Models. Support Llama-3/3.1, Llama-2, LLaMA, BLOOM, Vicuna, Baichuan, TinyLlama, etc.
PrismQuant: Optimal Null-Space Rotations for Grouped Quantizers