A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations
Repository Index
47,133 repositories indexed
1–24 of 26 · Most stars
RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.
- attention-mechanism
- deep-learning
- gpt
- gpt-2
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
Semantic cache for LLMs. Fully integrated with LangChain and llama_index.
TranslatedOpen-source manuscript of 《深入理解 AI Infra：量化分析与系统设计》 (In-Depth Understanding of AI Infra: Quantitative Analysis and System Design), by 李博杰 (Li Bojie): starting from hardware constraints and model architecture, it quantitatively derives the system design of LLM inference and training. Includes the full text of the book, a PDF, and companion calculation tools and experiments
- accelerator
- ai-infra
- ai-infrastructure
- book
🔩 Like builtins, but boltons. 250+ constructs, recipes, and snippets which extend (and rely on nothing but) the Python standard library. Nothing like Michael Bolton.
- python
- utilities
- cache
- data-structures
Python disk-backed cache (Django-compatible). Faster than Redis and Memcached. Pure-Python.
- python
- cache
- persistence
- key-value-store
Compresses linked and inline javascript or CSS into a single cached file.
Extensible memoizing collections and decorators
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
- kv-cache
- llm-inference
- local-llm
- quantization
A slick ORM cache with automatic granular event-driven invalidation.
Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.
Asyncio cache manager for redis, memcached and memory
TranslatedA Claude Code skill that digs into an unfamiliar field in a sweet-potato-vine fashion, one lead pulling up the next, until it reaches the original source text
30 different honeypots in one package! (dhcp, dns, elastic, ftp, http proxy, https proxy, http, https, imap, ipp, irc, ldap, memcache, mssql, mysql, ntp, oracle, pjl, pop3, postgres, rdp, redis, sip, smb, smtp, snmp, socks5, ssh, telnet, vnc)
- pypi
- monitoring-tool
- network-analysis
- malware-analysis
YAS3FS (Yet Another S3-backed File System) is a Filesystem in Userspace (FUSE) interface to Amazon S3. It was inspired by s3fs but rewritten from scratch to implement a distributed cache synchronized by Amazon SNS notifications. A web console is provided to easily monitor the nodes of a cluster.
Lightweight Claude Code statusLine: 5h/7d rate-limit usage, reset countdowns, model + context window, prompt-cache age — one line, 3 styles × 9 themes, daemon fast-mode
dogpile.cache is a Python caching API which provides a generic interface to caching backends of any variety
"Evolving AppCompat/AmCache data analysis beyond grep"
Official implementation of paper "Cache Engineering for Agents"
🐍 CLI tool to find and delete unused Python virtual environments (.venv, poetry, conda, pipenv, uv). Reclaim disk space effortlessly — supports bulk delete, size reporting, and pycache cleanup.
- command-line
- developer-tools
- python
- venv
A simple page cache for Wagtail based on the Django cache middleware.
Parses Java Cache IDX files
micropython SD card driver that works with LittleFS2 (implements extended interface)