Extensible memoizing collections and decorators
Repository Index
142,273 visible repositories in the index
25–48 of 119 · Most stars
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
- kv-cache
- llm-inference
- local-llm
- quantization
A runtime substrate that turns an agent's execution into a reversible, Git-like trace, so meta-agents can observe, fork, replay, and revert any run. Couples agent and environments in a copy-on-write fork ~5x faster than docker commit, with ~95% KV-cache reuse on replay. Framework built for meta-agents to supervise, optimize, and train other agents
- meta-agents
- meta-optimization
- runtime-supervision
- workflow-automation
Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.
A slick ORM cache with automatic granular event-driven invalidation.
An extendable async API using FastAPI, Pydantic V2, SQLAlchemy 2.0, PostgreSQL and Redis.
Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond
- gpu-sharing
- kvcache
- llm
- sglang
:superhero: :zap: django's default admin interface with superpowers - customizable themes, popup windows replaced by modals and many other features.
- django
- python
- admin
- custom
fastapi-cache is a tool to cache fastapi response and function result, with backends support redis and memcached.
TurboQuant: Near-optimal KV cache quantization for LLM inference (3-bit keys, 2-bit values) with Triton kernels + vLLM integration
Cache-Augmented Generation: A Simple, Efficient Alternative to RAG
:blue_book: dict subclass with keylist/keypath support, built-in I/O operations (base64, csv, html, ini, json, pickle, plist, query-string, toml, xls, xml, yaml), s3 support and many utilities.
- python
- dict
- dictionary
- keypath
Install and manage a high performance WordPress stack with a few keystrokes
Persistent HTTP cache for python requests
TranslatedA Claude Code skill that digs into an unfamiliar field in a sweet-potato-vine fashion, one lead pulling up the next, until it reaches the original source text
No effort, no worry, maximum performance.
Asyncio cache manager for redis, memcached and memory
DDoS attack tool for sending forged UDP packets to vulnerable Memcached servers obtained using Shodan API
Unified KV Cache Compression Methods for Auto-Regressive Models
- kv-cache
- kv-cache-compression
- llm
Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model
- diffusion-models
- inference-acceleration
- video-generation
- cogvideox
Zero-GPU CPU inference for 30B-284B MoE models (Qwen3.8 176.5B cold-cache certified in 28.26/48.35 GiB Cgroup v2, DeepSeek-V4 284B, Nemotron-30B) + RouteCrack 2.0.
A PyTorch-native inference engine with cache, parallelism, quantization and cpu offload for DiTs.
- flux2-klein
- svdquant
- flux3
- flux3-klein
This is a project template which uses FastAPI, Pydantic 2.0, Alembic and async SQLModel as ORM. It shows a complete async CRUD using authentication and role base access control.
- fastapi
- sqlmodel
- sqlalchemy
- alembic
LLM KV cache compression made easy