Tensor library for machine learning
Repository Index
102,118 visible repositories in the index
1–24 of 56 · Most stars
Easy-to-use and powerful LLM and SLM library with awesome model zoo.
- nlp
- embedding
- bert
- ernie
Accessible large language models via k-bit quantization for PyTorch.
- llm
- machine-learning
- pytorch
- qlora
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
- cuda-kernels
- deepspeed
- fastertransformer
- llm-inference
2bit quants, kernels and runtime for consumer devices.
A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.
The central nervous system and backend orchestrator for Fractal. It handles multi-tenant task scheduling, real-time TFLOPs budgeting, and federated weight aggregation using FedAvg.
A unified inference and post-training framework for accelerated video generation.
- diffusers
- diffusion-models
- video-generation
- distillation
Fast inference engine for Transformer models
- neural-machine-translation
- cpp
- mkl
- quantization
A fast inference library for running LLMs locally on modern consumer-class GPUs
A repository for storing models that have been inter-converted between various frameworks. Supported frameworks are TensorFlow, PyTorch, ONNX, OpenVINO, TFJS, TFTRT, TensorFlowLite (Float32/16/INT8), EdgeTPU, CoreML.
- tensorflow
- tensorflow-lite
- openvino
- edgetpu
State-of-the-art LLM compression, built for production inference with vLLM
- compression
- quantization
TurboDiffusion: 100–200× Acceleration for Video Diffusion Models
- ai-infra
- diffusion-models
- inference-acceleration
- rcm
[MLSys 2024 Best Paper Award] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.
🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools
llama.cpp fork with additional SOTA quants and improved performance
[CVPR 2023] DepGraph: Towards Any Structural Pruning; LLMs, Vision Foundation Models, etc.
- pruning
- model-compression
- efficient-deep-learning
- llm
4 bits quantization of LLaMA using GPTQ
PyTorch native quantization for training and inference
- brrr
- dtypes
- inference
- mx
An Automatic Model Compression (AutoMC) framework for developing smaller and faster AI applications.
- deep-learning
- model-compression
- mobile-app
- automl
An Open-source Toolkit for LLM Development
AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.
- quantization
- deep-learning
- compression
- open-source
SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime
- low-precision
- pruning
- sparsity
- auto-tuning