2bit quants, kernels and runtime for consumer devices.
Repository Index
142,273 visible repositories in the index
1–10 of 10 · Most stars
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
- kv-cache
- llm-inference
- local-llm
- quantization
TurboQuant: Near-optimal KV cache quantization for LLM inference (3-bit keys, 2-bit values) with Triton kernels + vLLM integration
Unified KV Cache Compression Methods for Auto-Regressive Models
- kv-cache
- kv-cache-compression
- llm
Zero-GPU CPU inference for 30B-284B MoE models (Qwen3.8 176.5B cold-cache certified in 28.26/48.35 GiB Cgroup v2, DeepSeek-V4 284B, Nemotron-30B) + RouteCrack 2.0.
LLM KV cache compression made easy
[Neurips 2025] R-KV: Redundancy-aware KV Cache Compression for Reasoning Models
- kvcache
- llm
- reasoning-models
KVarN, KV cache precision tail, low-bit quants in llama.cpp for longer context of better precision in the same VRAM
- ggml
- kv-cache
- llama-cpp
- llm-inference
KV cache compression via block-diagonal rotation. Beats TurboQuant: better PPL (6.91 vs 7.07), 28% faster decode, 5.3x faster prefill, 44x fewer params. Drop-in llama.cpp integration.
From-scratch PyTorch implementation of Google's TurboQuant (ICLR 2026) for LLM KV cache compression. 5x compression at 3-bit with 99.5% attention fidelity.