Repository Index

LLM inference in C/C++

• ggml

AirLLM 2.8T LLM inference with single 4GB GPU

• chinese-nlp
• finetune
• generative-ai
• instruct-gpt

Easy-to-use and powerful LLM and SLM library with awesome model zoo.

• nlp
• embedding
• bert
• ernie

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

• cuda-kernels
• deepspeed
• fastertransformer
• llm-inference

Run LLMs with MLX

• llms
• mlx

2bit quants, kernels and runtime for consumer devices.

• android
• framework
• ios
• llamacpp

Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.

A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.

The central nervous system and backend orchestrator for Fractal. It handles multi-tenant task scheduling, real-time TFLOPs budgeting, and federated weight aggregation using FedAvg.

• backend
• fedavg
• federated-learning
• flask

A unified inference and post-training framework for accelerated video generation.

• diffusers
• diffusion-models
• video-generation
• distillation

Fast inference engine for Transformer models

• neural-machine-translation
• cpp
• mkl
• quantization

A fast inference library for running LLMs locally on modern consumer-class GPUs

A repository for storing models that have been inter-converted between various frameworks. Supported frameworks are TensorFlow, PyTorch, ONNX, OpenVINO, TFJS, TFTRT, TensorFlowLite (Float32/16/INT8), EdgeTPU, CoreML.

• tensorflow
• tensorflow-lite
• openvino
• edgetpu

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

• kv-cache
• llm-inference
• local-llm
• quantization

🐋 Run 284B DeepSeek-V4-Flash (<59GiB RAM, 44.4 tok/s pp, 9.1-14.7 tok/s) & 30B MoEs (25-111 tok/s) on a pure 64GB CPU at 256K ctx (0 GPUs). SwiGLU-H2, Stockfish-PVS L3 & 13ms Claude Code restore.

• avx512
• cpu-inference
• hpc
• llama-cpp

QKeras: a quantization deep learning library for Tensorflow Keras

• deep-learning
• quantization
• quantized-neural-networks
• hardware-acceleration

Xolotl: run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM): ~33 tok/s on agent tasks, ~41 in fast mode, 512K context. Plus GLM-5.3-Flash, Qwen3.8 and LFM3-Omni. MoE experts tiered across VRAM, pinned RAM and NVMe, with Xolotl Direct NVMe->VRAM. OpenAI + Anthropic API.

• cuda
• deepseek
• deepseek-v4
• gguf

[Tech Report] Simplifying and Improving On-Policy Distillation for Few-step Diffusion

• diffusion
• distillation
• few-step-generation

PROMETHEVS: run the full DeepSeek-V4.1-Flash-1017 (510 GB) on a gaming PC with 64 GB RAM and a 12 GB GPU. PROMETHEVS engine (compress-then-correct), OpenAI-compatible local server, report and logs.

• deepseek
• llm
• local-inference
• mixture-of-experts

PrismQuant: Optimal Null-Space Rotations for Grouped Quantizers

• int4
• large-language-models
• pytorch
• quantization

This is the github repository for converting craft pretrained model to tflite version