Repository Index

Accessible large language models via k-bit quantization for PyTorch.

• llm
• machine-learning
• pytorch
• qlora

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

• kv-cache
• llm-inference
• local-llm
• quantization

🐋 Run 284B DeepSeek-V4-Flash (<59GiB RAM, 44.4 tok/s pp, 9.1-14.7 tok/s) & 30B MoEs (25-111 tok/s) on a pure 64GB CPU at 256K ctx (0 GPUs). SwiGLU-H2, Stockfish-PVS L3 & 13ms Claude Code restore.

• avx512
• cpu-inference
• hpc
• llama-cpp

QKeras: a quantization deep learning library for Tensorflow Keras

• deep-learning
• quantization
• quantized-neural-networks
• hardware-acceleration

Xolotl: run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM): ~33 tok/s on agent tasks, ~41 in fast mode, 512K context. Plus GLM-5.3-Flash, Qwen3.8 and LFM3-Omni. MoE experts tiered across VRAM, pinned RAM and NVMe, with Xolotl Direct NVMe->VRAM. OpenAI + Anthropic API.

• cuda
• deepseek
• deepseek-v4
• gguf

PROMETHEVS: run the full DeepSeek-V4.1-Flash-1017 (510 GB) on a gaming PC with 64 GB RAM and a 12 GB GPU. PROMETHEVS engine (compress-then-correct), OpenAI-compatible local server, report and logs.

• deepseek
• llm
• local-inference
• mixture-of-experts

PrismQuant: Optimal Null-Space Rotations for Grouped Quantizers

• int4
• large-language-models
• pytorch
• quantization