A high-performance inference engine for AI models
Repository Index
107,098 visible repositories in the index
73–96 of 238 · Most stars
Add a real-time analytics node to your operational database. Spice is a portable, accelerated SQL query, search, and LLM-inference engine in Rust for data-grounded AI apps and agents.
- artificial-intelligence
- developers
- machine-learning
- data
Distributed inference for mobile, desktop and server.
Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.
RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.
- ai
- containers
- cuda
- hip
Community maintained hardware plugin for vLLM on Huawei Ascend
Lucebox inference engine for heterogeneous hardware & consumer GPUs
High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.
Run GLM-5.3 (744B), GLM-5.3-Flash (321B), DeepSeek-V4.1-Flash (748B) and MiniMax-M3.1-Flash (428B) on your own GPUs - one engine, one dashboard, OpenAI- and Anthropic-compatible API.
TokenSpeed is a speed-of-light LLM inference engine.
Reference implementation for DPO (Direct Preference Optimization)
A frontend for running models on mobile or connecting to your preferred API providers.
The code used to train and run inference with the ColVision models, e.g. ColPali, ColQwen2, and ColSmol.
- colpali
- information-retrieval
- retrieval-augmented-generation
- vision-language-model
Mano-P: Open-source GUI-VLA agent for edge devices. #1 on OSWorld (specialized, 58.2%). Runs locally on Apple M4 Mac mini/MacBook — no data leaves your device.Mano-P 是一个开源 GUI-VLA 项目,支持在 Mac mini/MacBook 上或通过算力棒本地运行推理,实现纯视觉驱动的跨平台 GUI 自动化操作。数据完全本地处理,支持复杂多步骤任务规划与执行。
- computer-use-agents
- desktop-automation
- edge-computing
- gui-automation
A Unified Library for Parameter-Efficient and Modular Transfer Learning
- nlp
- natural-language-processing
- adapters
- transformers
vLLM’s reference system for K8S-native cluster-wide deployment with community-driven performance optimization
All-in-one platform for search, recommendations, RAG, and analytics offered via API
- actix-web
- diesel
- postgresql
- qdrant
The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in OpenCode on an M5 Max, and a 27B model on 16 GB Macs. Native MTP speculative decoding on Apple Silicon, exact at any temperature. OpenAI and Anthropic compatible local server.
- apple-silicon
- local-ai
- mlx
- mtp
The repository host gives no description.
- ai-sdk
- chat-completions
- foundry-local
- gpu-acceleration
Use Hugging Face with JavaScript
Run the full 2.78-trillion-parameter Kimi K3 model, DeepSeek V4.1 Flash or GLM-5.3-Flash beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine.
Native LLM inference server for Apple Silicon. OpenAI + Anthropic API compatible. No Python. Zig backend, Swift frontend macOS app with chat, music, voice, video generation.
- agent
- anthropic-api
- apple-silicon
- claude-code
LLM inference in C/C++
Manages Unified Access to Generative AI Services built on Envoy Gateway
- ai
- ai-gateway
- api-gateway
- kubernetes