Repository Index

Accessible large language models via k-bit quantization for PyTorch.

• llm
• machine-learning
• pytorch
• qlora

Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.

[ICLR2025, ICML2025, NeurIPS2025 Spotlight] Quantized Attention achieves speedup of 2-5x compared to FlashAttention, without losing end-to-end metrics across language, image, and video models.

• attention
• inference-acceleration
• llm
• quantization

Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.

• knowledge-distillation
• model-compression
• quantization
• pretrained-models

SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime

• low-precision
• pruning
• sparsity
• auto-tuning

Run Mixtral-8x7B models in Colab or consumer desktops

• colab-notebook
• deep-learning
• google-colab
• language-model

PrismQuant: Optimal Null-Space Rotations for Grouped Quantizers

• int4
• large-language-models
• pytorch
• quantization