Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.
Repository Index
132,241 visible repositories in the index
1–24 of 25 · Most stars
The central nervous system and backend orchestrator for Fractal. It handles multi-tenant task scheduling, real-time TFLOPs budgeting, and federated weight aggregation using FedAvg.
A repository for storing models that have been inter-converted between various frameworks. Supported frameworks are TensorFlow, PyTorch, ONNX, OpenVINO, TFJS, TFTRT, TensorFlowLite (Float32/16/INT8), EdgeTPU, CoreML.
- tensorflow
- tensorflow-lite
- openvino
- edgetpu
[ICLR2025, ICML2025, NeurIPS2025 Spotlight] Quantized Attention achieves speedup of 2-5x compared to FlashAttention, without losing end-to-end metrics across language, image, and video models.
- attention
- inference-acceleration
- llm
- quantization
A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.
🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers with easy to use hardware optimization tools
PyTorch native quantization for training and inference
- brrr
- dtypes
- inference
- mx
An Automatic Model Compression (AutoMC) framework for developing smaller and faster AI applications.
- deep-learning
- model-compression
- mobile-app
- automl
SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime
- low-precision
- pruning
- sparsity
- auto-tuning
Run Mixtral-8x7B models in Colab or consumer desktops
- colab-notebook
- deep-learning
- google-colab
- language-model
A PyTorch implementation for exploring deep and shallow knowledge distillation (KD) experiments with flexibility
- pytorch
- knowledge-distillation
- deep-neural-networks
- cifar10
OneDiff: An out-of-the-box acceleration library for diffusion models.
- comfyui
- diffusers
- pytorch
- sdxl
PPL Quantization Tool (PPQ) is a powerful offline neural network quantization tool.
- neural-network
- deep-learning
- quantization
- pytorch
Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.
- llm
- llm-compression
- quantization
- speculative-decoding
This repository provides code for machine learning algorithms for edge devices developed at Microsoft Research India.
- iot-device
- sensor
- edge-machine-learning
- resource-constrained-ml
A toolkit to optimize ML models for deployment for Keras and TensorFlow, including quantization and pruning.
Run 284B DeepSeek-V4-Flash (<59GiB RAM: 41.1 cold pp, 9.1 N=1 tg, up to 25.3 spec tok/s) & 30B MoEs (22.4 N=1 / 111.6 spec tok/s) on 64GB CPU (0 GPUs).
Neural Network Compression Framework for enhanced OpenVINO™ inference
- quantization
- pruning
- sparsity
- quantization-aware-training
NVIDIA FastGen: Fast Generation from Diffusion Models
- diffusion-models
- distillation
- consistency-models
- distribution-matching-distillation
Fast & Simple Resource-Constrained Learning of Deep Network Structure
- neural-architecture-search
- automl
- tensorflow
- deep-learning
YOLOv5 Series Multi-backbone(TPH-YOLOv5, Ghostnet, ShuffleNetv2, Mobilenetv3Small, EfficientNetLite, PP-LCNet, SwinTransformer YOLO), Module(CBAM, DCN), Pruning (EagleEye, Network Slimming), Quantization (MQBench) and Deployment (TensorRT, ncnn) Compression Tool Box.
- tph-yolov5
- ghostnet-yolov5
- shufflenetv2-yolov5
- mobilenetv3small-yolov5
QKeras: a quantization deep learning library for Tensorflow Keras
- deep-learning
- quantization
- quantized-neural-networks
- hardware-acceleration
Xolotl: run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM): ~33 tok/s on agent tasks, ~41 in fast mode, 512K context. Plus GLM-5.3-Flash, Qwen3.8 and LFM3-Omni. MoE experts tiered across VRAM, pinned RAM and NVMe, with Xolotl Direct NVMe->VRAM. OpenAI + Anthropic API.
PROMETHEVS: run the full DeepSeek-V4.1-Flash-1017 (510 GB) on a gaming PC with 64 GB RAM and a 12 GB GPU. PROMETHEVS engine (compress-then-correct), OpenAI-compatible local server, report and logs.