LLM inference in C/C++
- ggml
Repository Index
47,133 repositories indexed
1–23 of 23 · Most stars
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
- llm-inference
- ai-chat
Official inference framework for 1-bit LLMs
Distribute and run LLMs with a single file.
- cross-platform
- gguf
- llama-cpp
- local-ai
MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI.
- mnn
- convolution
- vulkan
- embedded-devices
A lightweight, lightning-fast, in-process vector database
A suite of WiFi/Bluetooth offensive and defensive tools for the ESP32
OpenVINO™ is an open source toolkit for optimizing and deploying AI inference
- inference
- deep-learning
- openvino
- ai
Production ready toolkit to run AI locally
Deeplake is AI Data Runtime for Agents. It provides serverless postgres with a multimodal datalake, enabling scalable retrieval and training.
QOwnNotes is a plain-text file notepad and todo-list manager with Markdown support and Nextcloud / ownCloud integration.
- owncloud
- c-plus-plus
- qownnotes
- nextcloud
The AI-native database built for LLM applications, providing incredibly fast hybrid search of dense vector, sparse vector, tensor (multi-vector), and full-text.
- ai-native
- nearest-neighbor-search
- rag
- vector-search
Open-source library of optimized deep learning operations (matmul, convolution, attention) for CPUs (x64, AArch64, RISC-V) and Intel GPUs. Used by PyTorch, TensorFlow, OpenVINO, and ONNX Runtime.
- onednn
- oneapi
- performance
- cpp
A local inference engine for Apple silicon, built around the model.
- apple-silicon
- coding-agents
- llm-inference
- macos
The Open Source Strix Halo AI engine. Qwen 3.8 Flash Next Q4_K_XL: 1,801.01 pp, 73.17 tg single user, 194.37 tok/s 8 users; Qwen 3.8 27B Q4_K_XL: 656.33pp, 70.56tg tok/s single user with DFlash2
- amd
- llm
- strix-halo
- gfx1151
Build AI agents that run 100% on-device. Sub-100ms latency on Qualcomm NPU. Zero cloud dependency.
- agent-framework
- ai-agent
- automotive
- edge-ai
Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Qwen3.6, Ornith 1.5, Gemma 4, Tiel Coder and MiniCPM-V on a second engine written for PolyStrata. One-click install for Windows and Linux, OpenAI/Anthropic API on localhost.
- anthropic-api
- cuda
- gemma
- gguf
llama.cpp on a jailbroken PS5: run any GGUF model natively on the console, starting with Qwen 3.8
Xolotl (formerly Strata-DSV4.1): run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 90 GB RAM) at up to ~27 tok/s, 512K context, October 1017 builds included. Plus GLM-5.3-Flash and Qwen3.8. MoE experts tiered across VRAM, pinned RAM and NVMe; OpenAI + Anthropic API.
Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.
A minimalistic C++ Jinja templating engine for LLM chat templates
Loopcut: an AI agent that works inside Blender. A fork of Blender (GPL).
llama.cpp fork that abliterates at inference