LLM inference in C/C++
- ggml
111,820 visible repositories in the index
1–24 of 100 · Most stars
LLM inference in C/C++
AirLLM 2.8T LLM inference with single 4GB GPU
Tensor library for machine learning
Enjoy the magic of Diffusion models!
Easy-to-use and powerful LLM and SLM library with awesome model zoo.
Accessible large language models via k-bit quantization for PyTorch.
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
Run LLMs with MLX
The repository host gives no description.
2bit quants, kernels and runtime for consumer devices.
Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.
PyTorch native post-training library
A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.
The central nervous system and backend orchestrator for Fractal. It handles multi-tenant task scheduling, real-time TFLOPs budgeting, and federated weight aggregation using FedAvg.
A unified inference and post-training framework for accelerated video generation.
Fast inference engine for Transformer models
A fast inference library for running LLMs locally on modern consumer-class GPUs
A repository for storing models that have been inter-converted between various frameworks. Supported frameworks are TensorFlow, PyTorch, ONNX, OpenVINO, TFJS, TFTRT, TensorFlowLite (Float32/16/INT8), EdgeTPU, CoreML.
[ICLR2025, ICML2025, NeurIPS2025 Spotlight] Quantized Attention achieves speedup of 2-5x compared to FlashAttention, without losing end-to-end metrics across language, image, and video models.
State-of-the-art LLM compression, built for production inference with vLLM
TurboDiffusion: 100–200× Acceleration for Video Diffusion Models
[MLSys 2024 Best Paper Award] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.
🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools