1.vllm-project / vllmA high-throughput and memory-efficient inference and serving engine for LLMsgptllmpytorchmodel-servingtransformerllm-serving+14 morePythonAPACHE-2.096K total24K forks+8,314 starsfrom 87,764 · score 9.03
2.LMCache / LMCacheLMCache: Supercharge Your LLM with the Fastest KV Cache Layeramdcudainferencekv-cachellmpytorch+4 morePythonAPACHE-2.012K total2.1K forks+1,324 starsfrom 10,932 · score 7.19