Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic.
- gateway
- generative-ai
- llm-inference
- llms
111,821 visible repositories in the index
121–144 of 872 · Most stars
Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic.
Native MLX runtime for Laya typed decision models — 7–14 ms short decisions on M3 Max. No text generation, PyTorch, or cloud API.
lightweight, standalone C++ inference engine for Google's Gemma models.
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
FlashInfer: Kernel Library for LLM Serving
LiteRT-LM is Google's production-ready, high-performance, open-source inference framework for deploying Large Language Models on edge devices.
[EMNLP'23, ACL'24] To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.
TranslatedDeveloper internal skills cultivation
Runtime type system for IO decoding/encoding
Modeling, training, eval, and inference code for OLMo
An open, programmable decision layer for models and compute.
On-device Speech AI for Apple Silicon
An anomaly detection library comprising state-of-the-art algorithms and features such as experiment management, hyper-parameter optimization, and edge inference.
Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams
Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes
Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk
2bit quants, kernels and runtime for consumer devices.
Awesome autocompletion, static analysis and refactoring library for python
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
Uplift modeling and causal inference with machine learning algorithms
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
:alarm_clock: AI conference deadline countdowns
A list of RSS related stuff: tools, services, communities and tutorials, etc.
A PHP static analysis tool for finding errors and security vulnerabilities in PHP applications