Repository Index

LLM inference in C/C++

• ggml

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

• llm-inference
• ai-chat

Official inference framework for 1-bit LLMs

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

Distribute and run LLMs with a single file.

• cross-platform
• gguf
• llama-cpp
• local-ai

Open-source library of optimized deep learning operations (matmul, convolution, attention) for CPUs (x64, AArch64, RISC-V) and Intel GPUs. Used by PyTorch, TensorFlow, OpenVINO, and ONNX Runtime.

• onednn
• oneapi
• performance
• cpp

Wifi & BLE driven passenger flow metering with cheap ESP32 boards

• wifi
• ttgo
• lopy
• heltec

A local inference engine for Apple silicon, built around the model.

• apple-silicon
• coding-agents
• llm-inference
• macos

Nearest Neighbor Search with Neighborhood Graph and Tree for High-dimensional Data

• approximate-nearest-neighbor-search
• k-nearest-neighbors
• knn-search
• nearest-neighbors

TranslatedUsing dense-compute cards to accelerate sparse, large-VRAM hosts, where the hosts include AI Max+ 395, M3 512G, DGX Spark, and others.

• llama-cpp
• pipeline-parallelism
• prefill-decode
• rtx-3060

The Open Source Strix Halo AI engine. Qwen 3.8 Flash Next Q4_K_XL: 1,801.01 pp, 73.17 tg single user, 194.37 tok/s 8 users; Qwen 3.8 27B Q4_K_XL: 656.33pp, 70.56tg tok/s single user with DFlash2

• amd
• llm
• strix-halo
• gfx1151

Embeddable, in-memory, document-oriented database with a high-level Query builder interface.

• database
• embedable
• go
• in-memory

Run large AI models on your own PC: Qwen3.8-Flash-Next on Strata's engine, and GLM 5.3 Flash, DeepSeek V4 Flash, LFM2.5, Qwen3.6, Ornith 1.5, Gemma 4, Tiel Coder and MiniCPM-V on a second engine written for PolyStrata. One-click install for Windows and Linux, OpenAI/Anthropic API on localhost.

• anthropic-api
• cuda
• gemma
• gguf

llama.cpp on a jailbroken PS5: run any GGUF model natively on the console, starting with Qwen 3.8

• homebrew
• llama-cpp
• llm
• playstation-5

Multi-Modal Database replacing MongoDB, Neo4J, and Elastic with 1 faster ACID solution, with NetworkX and Pandas interfaces, and bindings for C 99, C++ 17, Python 3, Java, GoLang 🗄️

• acid
• database
• key-value-store
• document-database

Blazing fast Apple Silicon inference engine optimized for Qwen & Nemotron. OpenAI + Anthropic API compatible. Built by folks from nace.ai

Local inference engine for Qwen3.8-Flash-Next on NVIDIA Blackwell

Xolotl (formerly Strata-DSV4.1): run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 90 GB RAM) at up to ~27 tok/s, 512K context, October 1017 builds included. Plus GLM-5.3-Flash and Qwen3.8. MoE experts tiered across VRAM, pinned RAM and NVMe; OpenAI + Anthropic API.

• cuda
• deepseek
• deepseek-v4
• gguf

Official library for RaBitQ vector quantization and approximate nearest-neighbor search, in C++ and Python.

• nearest-neighbor-search
• quantization
• vector-database
• vector-search

Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.

• apple-silicon
• cuda
• gguf
• inference

Fast fork of llama.cpp designed for Qwen3.8-Flash-Next

A minimalistic C++ Jinja templating engine for LLM chat templates

MobiInfer runs 35B-A3B-class MoE models fully on-device on the Qualcomm Hexagon NPU, streaming experts from flash with lookahead scheduling (LES) and DSpark speculation.

llama.cpp fork that abliterates at inference

• abliteration
• ai
• inference
• llama-cpp