Repository Index

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

• fine-tuning
• llama
• llm
• peft

Faster Whisper transcription with CTranslate2

• deep-learning
• inference
• quantization
• speech-recognition

TranslatedChinese LLaMA & Alpaca LLMs + local CPU/GPU training and deployment

• llm
• plm
• pre-trained-language-models
• alpaca

Accessible large language models via k-bit quantization for PyTorch.

• llm
• machine-learning
• pytorch
• qlora

Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

• kv-cache
• llm-inference
• local-llm
• quantization

Dataflow compiler for QNN inference on FPGAs

• dataflow
• quantization
• fpga
• compiler

🐋 Run 284B DeepSeek-V4-Flash (<59GiB RAM, 44.4 tok/s pp, 9.1-14.7 tok/s) & 30B MoEs (25-111 tok/s) on a pure 64GB CPU at 256K ctx (0 GPUs). SwiGLU-H2, Stockfish-PVS L3 & 13ms Claude Code restore.

• avx512
• cpu-inference
• hpc
• llama-cpp

QKeras: a quantization deep learning library for Tensorflow Keras

• deep-learning
• quantization
• quantized-neural-networks
• hardware-acceleration

Fast, exact inference for StartLux-Decision models on NVIDIA GPUs: the released GGUF files on int8 and 4-bit tensor cores, behind the /v1/systemone API, with up to 5.3× lower latency than vLLM in FP8 and bit-identical answers.

• calibration
• cuda
• cuda-graphs
• decision-making

PrismQuant: Optimal Null-Space Rotations for Grouped Quantizers

• int4
• large-language-models
• pytorch
• quantization