2bit quants, kernels and runtime for consumer devices.
Repository Index
111,820 visible repositories in the index
1–15 of 15 · Most stars
The central nervous system and backend orchestrator for Fractal. It handles multi-tenant task scheduling, real-time TFLOPs budgeting, and federated weight aggregation using FedAvg.
A fast inference library for running LLMs locally on modern consumer-class GPUs
TurboDiffusion: 100–200× Acceleration for Video Diffusion Models
- ai-infra
- diffusion-models
- inference-acceleration
- rcm
An Automatic Model Compression (AutoMC) framework for developing smaller and faster AI applications.
- deep-learning
- model-compression
- mobile-app
- automl
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
- kv-cache
- llm-inference
- local-llm
- quantization
Run Mixtral-8x7B models in Colab or consumer desktops
- colab-notebook
- deep-learning
- google-colab
- language-model
An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs
This repository provides code for machine learning algorithms for edge devices developed at Microsoft Research India.
- iot-device
- sensor
- edge-machine-learning
- resource-constrained-ml
A coding-free framework built on PyTorch for reproducible deep learning studies. PyTorch Ecosystem. 🏆28 knowledge distillation methods presented at TPAMI, CVPR, ICLR, ECCV, NeurIPS, ICCV, AAAI, etc are implemented so far. 🎁 Trained models, training logs and configurations are available for ensuring the reproducibiliy and benchmark.
- knowledge-distillation
- pytorch
- image-classification
- imagenet
This page is for the SlimYOLOv3: Narrower, Faster and Better for UAV Real-Time Applications
NVIDIA FastGen: Fast Generation from Diffusion Models
- diffusion-models
- distillation
- consistency-models
- distribution-matching-distillation
Fast & Simple Resource-Constrained Learning of Deep Network Structure
- neural-architecture-search
- automl
- tensorflow
- deep-learning
PROMETHEVS: run the full DeepSeek-V4.1-Flash-1017 (510 GB) on a gaming PC with 64 GB RAM and a 12 GB GPU. PROMETHEVS engine (compress-then-correct), OpenAI-compatible local server, report and logs.
PrismQuant: Optimal Null-Space Rotations for Grouped Quantizers
- int4
- large-language-models
- pytorch
- quantization