TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.
Repository Index
132,241 visible repositories in the index
25–48 of 324 · Most stars
20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
- llm
- llmops
- model-inference
- fine-tuning
LMCache: Supercharge Your LLM with the Fastest KV Cache Layer
The repository provides code for running inference and finetuning with the Meta Segment Anything Model 3 (SAM 3), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.
The Triton Inference Server provides an optimized cloud and edge inferencing solution.
Spark-TTS Inference Code
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
- bloom
- deep-learning
- distributed-systems
- language-models
PyTorch implementation of YOLOv3, YOLOv3-SPP, and YOLOv3-tiny for real-time object detection with training, validation, inference, and multi-format export.
- yolov3
- object-detection
- yolo
- deep-learning
Official Python inference and LoRA trainer package for the LTX-2 audio–video generative model.
- generative-ai
- ltx
- ltx-2
SANA & Sol: Efficient Generative Models and Full-Stack Training & Inference Engines for Image, Video, and World Modeling
- diffusion
- dit
- pytorch
- sana
Bayesian Modeling and Probabilistic Programming in Python
- python
- statistical-analysis
- bayesian-inference
- mcmc
Open-source inference platform for language, vision, embedding, reranking, image, audio, and video models. Deploy with multiple engines across heterogeneous hardware, from a laptop to a distributed cluster. Serve models through OpenAI-compatible APIs.
Easily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!
Deep universal probabilistic programming with Python and PyTorch
- python
- pytorch
- machine-learning
- bayesian
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
- model-serving
- mlops
- llmops
- generative-ai
AlphaFold 3 inference pipeline.
An Extensible Toolkit for Finetuning and Inference of Large Foundation Models. Large Models for All.
- chatgpt
- deep-learning
- instruction-following
- language-model
DoWhy is a Python library for causal inference that supports explicit modeling and testing of causal assumptions. DoWhy is based on a unified language for causal inference, combining causal graphical models and potential outcomes frameworks.
- causal-inference
- machine-learning
- graphical-models
- bayesian-networks
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
- cuda-kernels
- deepspeed
- fastertransformer
- llm-inference
TranslatedOpen-source manuscript of 《深入理解 AI Infra:量化分析与系统设计》 (In-Depth Understanding of AI Infra: Quantitative Analysis and System Design), by 李博杰 (Li Bojie): starting from hardware constraints and model architecture, it quantitatively derives the system design of LLM inference and training. Includes the full text of the book, a PDF, and companion calculation tools and experiments
- accelerator
- ai-infra
- ai-infrastructure
- book
TranslatedA collection of the Android SDK needed for Android development, tools used in development, Android development tutorials, Android design guidelines, free design resources, and more.
- android-sdk
- android-studio
- sdk-manager
- android-development
A framework for efficient model inference with omni-modality models
- diffusion
- inference
- model-serving
- pytorch
Open-source implementation of AlphaEvolve
- alphacode
- coding-agent
- deepmind
- deepmind-lab