DeepSpec: a full-stack codebase for training and evaluating speculative decoding algorithms
Repository Index
103,113 visible repositories in the index
1–16 of 16 · Most stars
DFlash: Block Diffusion for Flash Speculative Decoding
A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.
Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.
- apple-silicon
- local-llm
- macos
- mlx
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
- kv-cache
- llm-inference
- local-llm
- quantization
The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in OpenCode on an M5 Max, and a 27B model on 16 GB Macs. Native MTP speculative decoding on Apple Silicon, exact at any temperature. OpenAI and Anthropic compatible local server.
- apple-silicon
- local-ai
- mlx
- mtp
Official Implementation of EAGLE-1 (ICML'24), EAGLE-2 (EMNLP'24), and EAGLE-3 (NeurIPS'25).
- large-language-models
- llm-inference
- speculative-decoding
Large-scale LLM inference engine
- api-rest
- inference-engine
- machine-learning
- cuda
Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.
- llm
- llm-compression
- quantization
- speculative-decoding
฿ Bitcoin trading bot with a real-time dashboard for Bitstamp.
Run 284B DeepSeek-V4-Flash (<59GiB RAM: 41.1 cold pp, 9.1 N=1 tg, up to 25.3 spec tok/s) & 30B MoEs (22.4 N=1 / 111.6 spec tok/s) on 64GB CPU (0 GPUs).
Official Inference Code of EAGLE-4
- eagle
- gpu
- llm-inference
- sglang
Train speculative decoding models effortlessly and port them smoothly to SGLang serving.
- eagle
- eagle3
- pytorch
- sglang
Your iPhone helps your Mac run a 27B model: faster prompt reading and more context over a USB-C cable
A lightweight inference engine supporting speculative speculative decoding (SSD).
SpecPilot: Autonomous Algorithm Discovery and Systems Optimization for Speculative Decoding