LLM inference in C/C++
- ggml
111,085 visible repositories in the index
1–24 of 100 · Most stars
LLM inference in C/C++
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
Port of OpenAI's Whisper model in C/C++
Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
Official inference framework for 1-bit LLMs
Cross-platform, customizable ML solutions for live and streaming media.
SRS is a simple, high-performance, AI-driven real-time media server supporting RTMP, WebRTC, HLS, HTTP-FLV, HTTP-TS, SRT, MPEG-DASH, and GB28181, with codec support for H.264, H.265, AV1, VP9, AAC, Opus, and G.711.
Distribute and run LLMs with a single file.
ncnn is a high-performance neural network inference framework optimized for the mobile platform
Video, Image and GIF upscale/enlarge(Super-Resolution) and Video frame interpolation. Achieved with Waifu2x, Real-ESRGAN, Real-CUGAN, RTX Video Super Resolution VSR, SRMD, RealSR, Anime4K, RIFE, IFRNet, CAIN, DAIN, and ACNet.
MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI.
NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.
OpenVINO™ is an open source toolkit for optimizing and deploying AI inference
High-performance GPGPU inference of OpenAI's Whisper automatic speech recognition (ASR) model
A high-performance distributed file system designed to address the challenges of AI training and inference workloads.
Production ready toolkit to run AI locally
High-speed Large Language Model Serving for Local Deployment
Hello AI World guide to deploying deep-learning inference networks and deep vision primitives with TensorRT and NVIDIA Jetson.
Diffusion model(SD,Flux,Wan,Qwen Image,Z-Image,Minimax H3,...) inference in pure C/C++
Cutting Edge WebRTC Video Conferencing
TranslatedPaddlePaddle High Performance Deep Learning Inference Engine for Mobile and Edge
lightweight, standalone C++ inference engine for Google's Gemma models.
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
LiteRT-LM is Google's production-ready, high-performance, open-source inference framework for deploying Large Language Models on edge devices.