LLM inference in C/C++
- ggml
132,241 visible repositories in the index
1–24 of 106 · Most stars
LLM inference in C/C++
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
Port of OpenAI's Whisper model in C/C++
Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
Official inference framework for 1-bit LLMs
Cross-platform, customizable ML solutions for live and streaming media.
Distribute and run LLMs with a single file.
ncnn is a high-performance neural network inference framework optimized for the mobile platform
MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI.
NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.
OpenVINO™ is an open source toolkit for optimizing and deploying AI inference
High-performance GPGPU inference of OpenAI's Whisper automatic speech recognition (ASR) model
A high-performance distributed file system designed to address the challenges of AI training and inference workloads.
Production ready toolkit to run AI locally
High-speed Large Language Model Serving for Local Deployment
Hello AI World guide to deploying deep-learning inference networks and deep vision primitives with TensorRT and NVIDIA Jetson.
Diffusion model(SD,Flux,Wan,Qwen Image,Z-Image,Minimax H3,...) inference in pure C/C++
TranslatedPaddlePaddle High Performance Deep Learning Inference Engine for Mobile and Edge
lightweight, standalone C++ inference engine for Google's Gemma models.
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
LiteRT-LM is Google's production-ready, high-performance, open-source inference framework for deploying Large Language Models on edge devices.
Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk
2bit quants, kernels and runtime for consumer devices.
A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.