A high-throughput and memory-efficient inference and serving engine for LLMs
Repository Index
47,133 repositories indexed
1–8 of 8 · Most stars
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
- llm
- llmops
- model-inference
- fine-tuning
The AI Compute Platform for frontier teams. SkyPilot turns fragmented AI compute into one AI supercomputer, so frontier AI teams build custom intelligence faster.
- cloud-computing
- deep-learning
- gpu
- hyperparameter-tuning
《深入理解 AI Infra：量化分析与系统设计》（李博杰 著）开源书稿：从硬件约束和模型架构出发，量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验
- accelerator
- ai-infra
- ai-infrastructure
- book
Superduper: End-to-end framework for building custom AI applications and agents.
- ai
- mlops
- torch
- transformers
🧱 Experimental library for building inference engines