AirLLM 2.8T LLM inference with single 4GB GPU
- chinese-nlp
- finetune
- generative-ai
- instruct-gpt
Repository Index
82,088 visible repositories in the index
1–24 of 61 · Most stars
Enjoy the magic of Diffusion models!
Easy-to-use and powerful LLM and SLM library with awesome model zoo.
- nlp
- embedding
- bert
- ernie
Accessible large language models via k-bit quantization for PyTorch.
- llm
- machine-learning
- pytorch
- qlora
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
- cuda-kernels
- deepspeed
- fastertransformer
- llm-inference
Run LLMs with MLX
- llms
- mlx
The repository host gives no description.
Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.
PyTorch native post-training library
A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.
The central nervous system and backend orchestrator for Fractal. It handles multi-tenant task scheduling, real-time TFLOPs budgeting, and federated weight aggregation using FedAvg.
A unified inference and post-training framework for accelerated video generation.
- diffusers
- diffusion-models
- video-generation
- distillation
A fast inference library for running LLMs locally on modern consumer-class GPUs
A repository for storing models that have been inter-converted between various frameworks. Supported frameworks are TensorFlow, PyTorch, ONNX, OpenVINO, TFJS, TFTRT, TensorFlowLite (Float32/16/INT8), EdgeTPU, CoreML.
- tensorflow
- tensorflow-lite
- openvino
- edgetpu
State-of-the-art LLM compression, built for production inference with vLLM
- compression
- quantization
TurboDiffusion: 100–200× Acceleration for Video Diffusion Models
- ai-infra
- diffusion-models
- inference-acceleration
- rcm
[MLSys 2024 Best Paper Award] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.
🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools
[CVPR 2023] DepGraph: Towards Any Structural Pruning; LLMs, Vision Foundation Models, etc.
- pruning
- model-compression
- efficient-deep-learning
- llm
Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.
- knowledge-distillation
- model-compression
- quantization
- pretrained-models
4 bits quantization of LLaMA using GPTQ
PyTorch native quantization for training and inference
- brrr
- dtypes
- inference
- mx
An Automatic Model Compression (AutoMC) framework for developing smaller and faster AI applications.
- deep-learning
- model-compression
- mobile-app
- automl