AirLLM 2.8T LLM inference with single 4GB GPU
- chinese-nlp
- finetune
- generative-ai
- instruct-gpt
142,273 visible repositories in the index
1–24 of 74 · Most stars
AirLLM 2.8T LLM inference with single 4GB GPU
Easy-to-use and powerful LLM and SLM library with awesome model zoo.
Accessible large language models via k-bit quantization for PyTorch.
Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.
PyTorch native post-training library
A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.
The central nervous system and backend orchestrator for Fractal. It handles multi-tenant task scheduling, real-time TFLOPs budgeting, and federated weight aggregation using FedAvg.
A unified inference and post-training framework for accelerated video generation.
Fast inference engine for Transformer models
[ICLR2025, ICML2025, NeurIPS2025 Spotlight] Quantized Attention achieves speedup of 2-5x compared to FlashAttention, without losing end-to-end metrics across language, image, and video models.
State-of-the-art LLM compression, built for production inference with vLLM
TurboDiffusion: 100–200× Acceleration for Video Diffusion Models
[MLSys 2024 Best Paper Award] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.
🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools
[CVPR 2023] DepGraph: Towards Any Structural Pruning; LLMs, Vision Foundation Models, etc.
Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.
4 bits quantization of LLaMA using GPTQ
PyTorch native quantization for training and inference
An Automatic Model Compression (AutoMC) framework for developing smaller and faster AI applications.
Images to inference with no labeling (use foundation models to train supervised models).
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.
SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime