Pure C++ implementation of several models for real-time chatting on your computer (CPU & GPU)
Repository Index
132,241 visible repositories in the index
73–96 of 106 · Most stars
A uniform interface to run deep learning models from multiple frameworks
BladeDISC is an end-to-end DynamIc Shape Compiler project for machine learning workloads.
Xbox 360 to PC port of Dante's Inferno using ReXGlue (static recompilation)
Experimental llama.cpp fork for inference research and development
A highly optimized LLM inference acceleration engine for Llama and its variants.
Suno AI's Bark model in C/C++ for fast text-to-speech generation
Real-time neural network inferencing
Blazing fast Apple Silicon inference engine optimized for Qwen & Nemotron. OpenAI + Anthropic API compatible. Built by folks from nace.ai
- apple-silicon
- clover
- coding-agents
- decision-models
FlashRT is a high-performance realtime inference engine for small-batch, latency-sensitive AI workloads. The flagship integration is production VLA control for Pi0, Pi0.5, GROOT N1.6, and Pi0-FAST. Also support llm e.g, qwen3.6-27B
- cuda
- cuda-kernels
- realtime-inference
- realtime-vla
TensorRT C++ API Tutorial
Adlik: Toolkit for Accelerating Deep Learning Inference
- deep-learning
- inference
- tensorflow-serving
- openvino
Run TensorFlow models in C++ without installation and without Bazel
- cpp
- tensorflow
- inference
- c
AI Inferencing at the Edge. A simple one-file way to run various GGML models with KoboldAI's UI with AMD ROCm offloading
dabnn is an accelerated binary neural networks inference framework for mobile platform
The Tensor Algebra SuperOptimizer for Deep Learning
- deep-learning
- deep-neural-networks
- inference-optimization
🔥 (yolov3 yolov4 yolov5 unet ...)A mini pytorch inference framework which inspired from darknet.
- yolov3
- yolov4
- yolov5
- pytorch
Deep neural network library and toolkit to do high performace inference on NVIDIA jetson platforms
The repository host gives no description.
Triton Python, C++ and Java client libraries, and GRPC-generated client examples for go, java and scala.
Triton backend that enables pre-process, post-processing and other logic to be implemented in Python.
oneAPI Data Analytics Library (oneDAL)
- onedal
- data-analysis
- oneapi
- machine-learning
Deep Learning Streamer (DL Streamer) Pipeline Framework is an open-source streaming media analytics framework, based on GStreamer* multimedia framework, for creating complex media analytics pipelines for the Cloud or at the Edge.
- gstreamer
- gstreamer-plugins
- inference
- intel
Xolotl: run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM): ~33 tok/s on agent tasks, ~41 in fast mode, 512K context. Plus GLM-5.3-Flash, Qwen3.8 and LFM3-Omni. MoE experts tiered across VRAM, pinned RAM and NVMe, with Xolotl Direct NVMe->VRAM. OpenAI + Anthropic API.