DeepSpec: a full-stack codebase for training and evaluating speculative decoding algorithms
Repository Index
103,113 visible repositories in the index
1–24 of 35 · Most stars
DFlash: Block Diffusion for Flash Speculative Decoding
A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.
Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.
- apple-silicon
- local-llm
- macos
- mlx
Lucebox inference engine for heterogeneous hardware & consumer GPUs
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
- kv-cache
- llm-inference
- local-llm
- quantization
The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in OpenCode on an M5 Max, and a 27B model on 16 GB Macs. Native MTP speculative decoding on Apple Silicon, exact at any temperature. OpenAI and Anthropic compatible local server.
- apple-silicon
- local-ai
- mlx
- mtp
Official Implementation of EAGLE-1 (ICML'24), EAGLE-2 (EMNLP'24), and EAGLE-3 (NeurIPS'25).
- large-language-models
- llm-inference
- speculative-decoding
A local inference engine for Apple silicon, built around the model.
- apple-silicon
- coding-agents
- llm-inference
- macos
Large-scale LLM inference engine
- api-rest
- inference-engine
- machine-learning
- cuda
Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.
- llm
- llm-compression
- quantization
- speculative-decoding
AwesomeBump is a free program written using Qt library designed to generate normal, height, specular or ambient occlusion textures from a single image. Since the image processing is done in 99% on GPU the program runs very fast and all the parameters can be changed in real time. AB was made to be a new alternative to known gimp plugin called Insane Bump.
An interactive, explorable explanation about the peculiar magic of sound waves.
฿ Bitcoin trading bot with a real-time dashboard for Bitstamp.
PKI.js is a pure JavaScript library implementing the formats that are used in PKI applications (signing, encryption, certificate requests, OCSP and TSP requests/responses). It is built on WebCrypto (Web Cryptography API) and requires no plug-ins.
- pki
- webcrypto
- pki-applications
- javascript-library
Local inference that behaves like a frontier API: Qwen3.8-Flash-Next on AMD Strix Halo plus small NPU models, over the OpenAI and Anthropic APIs on one port.
- amd
- gfx1151
- gpu-inference
- hip
Run 284B DeepSeek-V4-Flash (<59GiB RAM: 41.1 cold pp, 9.1 N=1 tg, up to 25.3 spec tok/s) & 30B MoEs (22.4 N=1 / 111.6 spec tok/s) on 64GB CPU (0 GPUs).
📰 Must-read papers and blogs on Speculative Decoding ⚡️
Official Inference Code of EAGLE-4
- eagle
- gpu
- llm-inference
- sglang
Train speculative decoding models effortlessly and port them smoothly to SGLang serving.
- eagle
- eagle3
- pytorch
- sglang
A 10-week, 30-minutes-a-day roadmap for LLM inference serving and optimization. vLLM, SGLang, quantization, speculative decoding, benchmarking.
- learning-resources
- llm
- llm-inference
- mlops
Your iPhone helps your Mac run a 27B model: faster prompt reading and more context over a USB-C cable
Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Qwen3.6, Ornith 1.5, Gemma 4, Tiel Coder and MiniCPM-V on a second engine written for PolyStrata. One-click install for Windows and Linux, OpenAI/Anthropic API on localhost.
- anthropic-api
- cuda
- gemma
- gguf
A lightweight inference engine supporting speculative speculative decoding (SSD).