AirLLM 2.8T LLM inference with single 4GB GPU
- chinese-nlp
- finetune
- generative-ai
- instruct-gpt
Repository Index
61,121 visible repositories in the index
1–18 of 18 · Most stars
Tensor library for machine learning
Easy-to-use and powerful LLM and SLM library with awesome model zoo.
- nlp
- embedding
- bert
- ernie
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
- cuda-kernels
- deepspeed
- fastertransformer
- llm-inference
2bit quants, kernels and runtime for consumer devices.
Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.
A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.
The central nervous system and backend orchestrator for Fractal. It handles multi-tenant task scheduling, real-time TFLOPs budgeting, and federated weight aggregation using FedAvg.
Fast inference engine for Transformer models
- neural-machine-translation
- cpp
- mkl
- quantization
A fast inference library for running LLMs locally on modern consumer-class GPUs
A repository for storing models that have been inter-converted between various frameworks. Supported frameworks are TensorFlow, PyTorch, ONNX, OpenVINO, TFJS, TFTRT, TensorFlowLite (Float32/16/INT8), EdgeTPU, CoreML.
- tensorflow
- tensorflow-lite
- openvino
- edgetpu
TurboDiffusion: 100–200× Acceleration for Video Diffusion Models
- ai-infra
- diffusion-models
- inference-acceleration
- rcm
A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
- kv-cache
- llm-inference
- local-llm
- quantization
QKeras: a quantization deep learning library for Tensorflow Keras
- deep-learning
- quantization
- quantized-neural-networks
- hardware-acceleration
Xolotl: run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM): ~33 tok/s on agent tasks, ~41 in fast mode, 512K context. Plus GLM-5.3-Flash, Qwen3.8 and LFM3-Omni. MoE experts tiered across VRAM, pinned RAM and NVMe, with Xolotl Direct NVMe->VRAM. OpenAI + Anthropic API.
PROMETHEVS: run the full DeepSeek-V4.1-Flash-1017 (510 GB) on a gaming PC with 64 GB RAM and a 12 GB GPU. PROMETHEVS engine (compress-then-correct), OpenAI-compatible local server, report and logs.
PrismQuant: Optimal Null-Space Rotations for Grouped Quantizers
- int4
- large-language-models
- pytorch
- quantization