1.vllm-project / vllmA high-throughput and memory-efficient inference and serving engine for LLMsgptllmpytorchmodel-servingtransformerllm-serving+14 morePythonAPACHE-2.096K total24K forks+3,124 starsfrom 92,954 · score 8.05
2.LMCache / LMCacheLMCache: Supercharge Your LLM with the Fastest KV Cache Layeramdcudainferencekv-cachellmpytorch+4 morePythonAPACHE-2.012K total2.1K forks+335 starsfrom 11,921 · score 5.82