Repository Index

LLM inference in C/C++

• ggml

AirLLM 2.8T LLM inference with single 4GB GPU

• chinese-nlp
• finetune
• generative-ai
• instruct-gpt

Tensor library for machine learning

• machine-learning

Enjoy the magic of Diffusion models!

Easy-to-use and powerful LLM and SLM library with awesome model zoo.

• nlp
• embedding
• bert
• ernie

Accessible large language models via k-bit quantization for PyTorch.

• llm
• machine-learning
• pytorch
• qlora

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

• cuda-kernels
• deepspeed
• fastertransformer
• llm-inference

The repository host gives no description.

2bit quants, kernels and runtime for consumer devices.

• android
• framework
• ios
• llamacpp

Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.

PyTorch native post-training library

A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.

The central nervous system and backend orchestrator for Fractal. It handles multi-tenant task scheduling, real-time TFLOPs budgeting, and federated weight aggregation using FedAvg.

• backend
• fedavg
• federated-learning
• flask

A unified inference and post-training framework for accelerated video generation.

• diffusers
• diffusion-models
• video-generation
• distillation

Fast inference engine for Transformer models

• neural-machine-translation
• cpp
• mkl
• quantization

A fast inference library for running LLMs locally on modern consumer-class GPUs

A repository for storing models that have been inter-converted between various frameworks. Supported frameworks are TensorFlow, PyTorch, ONNX, OpenVINO, TFJS, TFTRT, TensorFlowLite (Float32/16/INT8), EdgeTPU, CoreML.

• tensorflow
• tensorflow-lite
• openvino
• edgetpu

[ICLR2025, ICML2025, NeurIPS2025 Spotlight] Quantized Attention achieves speedup of 2-5x compared to FlashAttention, without losing end-to-end metrics across language, image, and video models.

• attention
• inference-acceleration
• llm
• quantization

State-of-the-art LLM compression, built for production inference with vLLM

• compression
• quantization

TurboDiffusion: 100–200× Acceleration for Video Diffusion Models

• ai-infra
• diffusion-models
• inference-acceleration
• rcm

Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.

• kv-cache
• llm-inference
• local-llm
• quantization

Run 284B DeepSeek-V4-Flash (<59GiB RAM: 41.1 cold pp, 9.1 N=1 tg, up to 25.3 spec tok/s) & 30B MoEs (22.4 N=1 / 111.6 spec tok/s) on 64GB CPU (0 GPUs).

• avx512
• cpu-inference
• hpc
• llama-cpp

OmniEdu: Open Foundation Models for Learning and Teaching

QKeras: a quantization deep learning library for Tensorflow Keras

• deep-learning
• quantization
• quantized-neural-networks
• hardware-acceleration