LLM inference in C/C++
- ggml
Repository Index
47,133 repositories indexed
1–24 of 29 · Most stars
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
- llm-inference
- ai-chat
Port of OpenAI's Whisper model in C/C++
Official inference framework for 1-bit LLMs
Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
Cross-platform, customizable ML solutions for live and streaming media.
- mediapipe
- c-plus-plus
- computer-vision
- deep-learning
Distribute and run LLMs with a single file.
- cross-platform
- gguf
- llama-cpp
- local-ai
ncnn is a high-performance neural network inference framework optimized for the mobile platform
MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI.
- mnn
- convolution
- vulkan
- embedded-devices
NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.
- tensorrt
- nvidia
- deep-learning
- inference
OpenVINO™ is an open source toolkit for optimizing and deploying AI inference
- inference
- deep-learning
- openvino
- ai
High-performance GPGPU inference of OpenAI's Whisper automatic speech recognition (ASR) model
A high-performance distributed file system designed to address the challenges of AI training and inference workloads.
- distributed-file-system
Production ready toolkit to run AI locally
Open-source library of optimized deep learning operations (matmul, convolution, attention) for CPUs (x64, AArch64, RISC-V) and Intel GPUs. Used by PyTorch, TensorFlow, OpenVINO, and ONNX Runtime.
- onednn
- oneapi
- performance
- cpp
A local inference engine for Apple silicon, built around the model.
- apple-silicon
- coding-agents
- llm-inference
- macos
Deep learning inference nodes for ROS / ROS2 with support for NVIDIA Jetson and TensorRT
A uniform interface to run deep learning models from multiple frameworks
Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Qwen3.6, Ornith 1.5, Gemma 4, Tiel Coder and MiniCPM-V on a second engine written for PolyStrata. One-click install for Windows and Linux, OpenAI/Anthropic API on localhost.
- anthropic-api
- cuda
- gemma
- gguf
oneAPI Data Analytics Library (oneDAL)
- onedal
- data-analysis
- oneapi
- machine-learning
Blazing fast Apple Silicon inference engine optimized for Qwen & Nemotron. OpenAI + Anthropic API compatible. Built by folks from nace.ai
Local inference engine for Qwen3.8-Flash-Next on NVIDIA Blackwell
Xolotl (formerly Strata-DSV4.1): run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 90 GB RAM) at up to ~27 tok/s, 512K context, October 1017 builds included. Plus GLM-5.3-Flash and Qwen3.8. MoE experts tiered across VRAM, pinned RAM and NVMe; OpenAI + Anthropic API.
Inference module for RangeNet++ (milioto2019iros, chen2019iros)