A high-throughput and memory-efficient inference and serving engine for LLMs
Repository Index
47,133 repositories indexed
1–23 of 23 · Most stars
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
SGLang is a high-performance serving framework for large language models and multimodal models.
Open-source speech recognition toolkit for training, inference, streaming ASR, VAD, punctuation, speaker diarization pipelines, and OpenAI-compatible/MCP serving.
- pytorch
- speech-recognition
- paraformer
- punctuation
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
- llm
- llmops
- model-inference
- fine-tuning
InstantID: Zero-shot Identity-Preserving Generation in Seconds 🔥
The AI Compute Platform for frontier teams. SkyPilot turns fragmented AI compute into one AI supercomputer, so frontier AI teams build custom intelligence faster.
- cloud-computing
- deep-learning
- gpu
- hyperparameter-tuning
TranslatedOpen-source manuscript of 《深入理解 AI Infra：量化分析与系统设计》 (In-Depth Understanding of AI Infra: Quantitative Analysis and System Design), by 李博杰 (Li Bojie): starting from hardware constraints and model architecture, it quantitatively derives the system design of LLM inference and training. Includes the full text of the book, a PDF, and companion calculation tools and experiments
- accelerator
- ai-infra
- ai-infrastructure
- book
Fawkes, privacy preserving tool against facial recognition systems. More info at https://sandlab.cs.uchicago.edu/fawkes
- privacy-protection
- privacy-enhancing-technologies
- face-recognition
- adversarial-machine-learning
Superduper: End-to-end framework for building custom AI applications and agents.
- ai
- mlops
- torch
- transformers
Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.
- ai-gateway
- observability
- simulation
- ai-agents
Python package for earth-observing satellite data processing
- python
- satellite
- weather
- hacktoberfest
Nintendo Switch library manager, with automated management tasks, serving your library to multiple supported clients on your Switch, with shop customization and multi user authentication.
- tinfoil
- atmosphere
- homebrew
- switch
Native vLLM serving for Jev decision models
Tools for downloading and preserving wikis. We archive wikis, from Wikipedia to tiniest wikis. As of 2026, WikiTeam has preserved more than 600,000 wikis.
- wiki
- wikiteam
- wikipedia
- archive-wikis
A collection of state-of-the-art algorithms for the training, serving and interpretation of Decision Forest models in Keras.
- python
- machine-learning
- random-forest
- tensorflow
🧱 Experimental library for building inference engines
An evidence-preserving Chinese-to-English manuscript translation and review workflow for computing, medicine and other STEM disciplines.
Reference-preserving sprite animation skill for AI agents: intact-limb walks, small jumps, transparent PNG sequences and GIF.
- codex-skill
- pixel-art
- python
- sprite-animation
The backend serving https://stats.golem.network with data. It's available at https://api.stats.golem.network
- golem
Simple MicroPython web server for serving a website from a captive portal