AirLLM 2.8T LLM inference with single 4GB GPU
- chinese-nlp
- finetune
- generative-ai
- instruct-gpt
Repository Index
81,588 visible repositories in the index
1–18 of 18 · Most stars
Easy-to-use and powerful LLM and SLM library with awesome model zoo.
- nlp
- embedding
- bert
- ernie
2bit quants, kernels and runtime for consumer devices.
A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.
The central nervous system and backend orchestrator for Fractal. It handles multi-tenant task scheduling, real-time TFLOPs budgeting, and federated weight aggregation using FedAvg.
Fast inference engine for Transformer models
- neural-machine-translation
- cpp
- mkl
- quantization
State-of-the-art LLM compression, built for production inference with vLLM
- compression
- quantization
[MLSys 2024 Best Paper Award] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
[CVPR 2023] DepGraph: Towards Any Structural Pruning; LLMs, Vision Foundation Models, etc.
- pruning
- model-compression
- efficient-deep-learning
- llm
Images to inference with no labeling (use foundation models to train supervised models).
- computer-vision
- model-distillation
- auto-labeling
- deep-learning
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
- kv-cache
- llm-inference
- local-llm
- quantization
micronet, a model compression and deploy lib. compression: 1、quantization: quantization-aware-training(QAT), High-Bit(>2b)(DoReFa/Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference)、Low-Bit(≤2b)/Ternary and Binary(TWN/BNN/XNOR-Net); post-training-quantization(PTQ), 8-bit(tensorrt); 2、 pruning: normal、regular and group convolutional channel pruning; 3、 group convolution structure; 4、batch-normalization fuse for quantization. deploy: tensorrt, fp32/fp16/int8(ptq-calibration)、op-adapt(upsample)、dynamic_shape
- quantization
- pruning
- dorefa
- twn
OpenMMLab Model Compression Toolbox and Benchmark.
- nas
- pruning
- knowledge-distillation
- spos
在 oxford hand 数据集上对 YOLOv3 做模型剪枝（network slimming）
- yolov3
- model-pruning
- hand-detection
- channel-pruning
A coding-free framework built on PyTorch for reproducible deep learning studies. PyTorch Ecosystem. 🏆28 knowledge distillation methods presented at TPAMI, CVPR, ICLR, ECCV, NeurIPS, ICCV, AAAI, etc are implemented so far. 🎁 Trained models, training logs and configurations are available for ensuring the reproducibiliy and benchmark.
- knowledge-distillation
- pytorch
- image-classification
- imagenet
PaddleSlim is an open-source library for deep model compression and architecture search.
- pruning
- quantization
- nas
- bert
Xolotl: run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM): ~33 tok/s on agent tasks, ~41 in fast mode, 512K context. Plus GLM-5.3-Flash, Qwen3.8 and LFM3-Omni. MoE experts tiered across VRAM, pinned RAM and NVMe, with Xolotl Direct NVMe->VRAM. OpenAI + Anthropic API.
PROMETHEVS: run the full DeepSeek-V4.1-Flash-1017 (510 GB) on a gaming PC with 64 GB RAM and a 12 GB GPU. PROMETHEVS engine (compress-then-correct), OpenAI-compatible local server, report and logs.