TranslatedPaddlePaddle High Performance Deep Learning Inference Engine for Mobile and Edge
Repository Index
132,241 visible repositories in the index
97–120 of 748 · Most stars
Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic.
- gateway
- generative-ai
- llm-inference
- llms
Native MLX runtime for Laya typed decision models — 7–14 ms short decisions on M3 Max. No text generation, PyTorch, or cloud API.
- apple-silicon
- inference
- machine-learning
- mlx
lightweight, standalone C++ inference engine for Google's Gemma models.
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
FlashInfer: Kernel Library for LLM Serving
LiteRT-LM is Google's production-ready, high-performance, open-source inference framework for deploying Large Language Models on edge devices.
- edge-ai
- on-device-ai
- on-device-llm
[EMNLP'23, ACL'24] To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.
Runtime type system for IO decoding/encoding
- typescript
- validation
- inference
- types
Modeling, training, eval, and inference code for OLMo
An open, programmable decision layer for models and compute.
- mixture-of-models
- semantic-router
- vllm
- ai-gateway
On-device Speech AI for Apple Silicon
- inference
- ios
- speech-recognition
- swift
An anomaly detection library comprising state-of-the-art algorithms and features such as experiment management, hyper-parameter optimization, and edge inference.
- anomaly-localization
- anomaly-segmentation
- anomaly-detection
- unsupervised-learning
Adversarial Robustness Toolbox (ART) - Python Library for Machine Learning Security - Evasion, Poisoning, Extraction, Inference - Red and Blue Teams
- python
- attack
- adversarial-machine-learning
- poisoning
Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk
- amd
- llama
- llm
- llm-inference
Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes
- knative
- machine-learning
- model-interpretability
- model-serving
2bit quants, kernels and runtime for consumer devices.
Awesome autocompletion, static analysis and refactoring library for python
- static-analysis
- auto-complete
- python
- refactoring
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
Uplift modeling and causal inference with machine learning algorithms
- incubation
- machine-learning
- causal-inference
- uplift-modeling
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
- llama
- llamacpp
- llm-inference
- ollama-api
A PHP static analysis tool for finding errors and security vulnerabilities in PHP applications
- static-analysis
- php
- type-inference
- security-analysis
Implement a reasoning LLM in PyTorch from scratch, step by step
MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac using MLX.
- llava
- llm
- mlx
- vision-transformer