A scalable inference server for models optimized with OpenVINO™
Repository Index
132,241 visible repositories in the index
145–168 of 233 · Most stars
An open-source framework for verifiably private AI inference
Perplexity open source garden for inference technology
The Triton TensorRT-LLM Backend
Pure C++ implementation of several models for real-time chatting on your computer (CPU & GPU)
LeaderWorkerSet & DisaggregatedSet: Kubernetes APIs for Distributed and Disaggregated AI Serving
- sig-apps
- llm-inference
A highly optimized LLM inference acceleration engine for Llama and its variants.
LLM notes, including model inference, transformer model structure, and llm framework code analysis notes.
- llm
- llm-inference
- triton-kernels
- vllm
Android 17 local LLM prototype with Jetpack Compose and ONNX Runtime for offline AI inference experiments.
- android
- android-17
- edge-ai
- jetpack-compose
List of papers related to neural network quantization in recent AI conferences and journals.
- quantization
- awesome-list
- papers
- neural-networks
The most atomic way to train and inference a GPT in pure, dependency-free C
- artificial-intelligence
- c
- c-programming
- deep-learning
Omega-AI is a Java-based deep learning framework that helps you quickly build neural networks for inference and training. Its engine supports automatic differentiation, multithreading, and GPU acceleration with CUDA and cuDNN.
- llm
- yolo
- deeplearning
- diffusion
Blazing fast Apple Silicon inference engine optimized for Qwen & Nemotron. OpenAI + Anthropic API compatible. Built by folks from nace.ai
- apple-silicon
- clover
- coding-agents
- decision-models
🐈 A collection of LLM inference providers and models
The repository host gives no description.
RDNA-native LLM inference engine in Rust.
- amd-gpu
- gpu-computing
- hip
- llm-inference
Llama 3+ inference in pure Java
FlashRT is a high-performance realtime inference engine for small-batch, latency-sensitive AI workloads. The flagship integration is production VLA control for Pi0, Pi0.5, GROOT N1.6, and Pi0-FAST. Also support llm e.g, qwen3.6-27B
- cuda
- cuda-kernels
- realtime-inference
- realtime-vla
Frontier-class open models on a free Kaggle TPU v5e-8: GLM-5.3-Flash 320B MoE (~165 tok/s, our own JAX engine) and Qwen3.8-27B bf16 (~130 tok/s), 262k context, prefix caching. Works with Claude Code, Codex, opencode and pi.
- claude-code
- coding-agents
- free-gpu
- jax
An open-source solution for full parameter fine-tuning of DeepSeek-V3/R1 671B, including complete code and scripts from training to inference, as well as some practical experiences and conclusions. (DeepSeek-V3/R1 满血版 671B 全参数微调的开源解决方案,包含从训练到推理的完整代码和脚本,以及实践中积累一些经验和结论。)
- deepseek-r1
- llm
- moe
- sft
Hands-on LLM algorithms and systems with PyTorch, Triton & CUDA.《大模型算法与系统教程》:模型架构与训练、后训练、推理加速、性能与算子优化,配套 Notebook 练习、答案解析、测试与实验。
Free AI coding environment: Ollama + Python
- coding-assistant
- local-inference
- ollama
⚡ Native MLX Swift LLM inference server for Apple Silicon. OpenAI-compatible API, SSD streaming for 100B+ MoE models, TurboQuant KV cache compression, MACOS + iOS iPhone app.
A Python library for simulation-based inference with deep learning
- deep-learning
- amortized-inference
- bayesian-statistics
- computational-modeling