1.vllm-project / vllmA high-throughput and memory-efficient inference and serving engine for LLMsgptllmpytorchmodel-servingtransformerllm-serving+14 morePythonAPACHE-2.096K total24K forks+17,617 starsfrom 78,461 · score 9.78
2.LMCache / LMCacheLMCache: Supercharge Your LLM with the Fastest KV Cache Layeramdcudainferencekv-cachellmpytorch+4 morePythonAPACHE-2.012K total2.1K forks+4,169 starsfrom 8,087 · score 8.34