A native .NET LLM inference engine and agent runtime for GGUF models. TensorSharp provides a console application, a web-based chatbot interface, iPhone App, and Ollama/OpenAI-compatible HTTP APIs for programmatic access. It supports Windows/MacOS/iOS/Linux with full GPU capability
Repository Index
132,241 visible repositories in the index
193–216 of 233 · Most stars
Go with your own intelligence - Write Go applications that directly integrate llama.cpp for local inference using hardware acceleration on Linux, macOS, Windows, & WebAssembly.
Run a 105 GB AI model on a Mac that can't hold it. Qwen3.8-Flash-Next 4-bit, streamed from SSD on 16–64 GB Apple Silicon Macs. Fully native Swift, MLX and Metal. Offline, no Python. CLI, Swift library, and Ollama/OpenAI/Anthropic APIs. Works with Claude Code and Codex.
- apple-silicon
- llm
- llm-inference
- local-llm
DFloat11 [NeurIPS '25]: Lossless Compression of LLMs and DiTs for Efficient GPU Inference
LLaSA: Scaling Train-time and Inference-time Compute for LLaMA-based Speech Synthesis
- llama
- llm
- text-to-speech
- xcodec2
Peer-to-peer LLM inference in the browser: pool your devices to run big open models, for chat and coding agents.
Turn your Android phone into an OpenAI-compatible LLM inference server - Fully local, private and Open Source
- kotlin-android
- litert
- litert-lm
- llm
High-performance, light-weight C++ LLM and VLM Inference Software for Physical AI
🤗 Optimum Intel: Accelerate inference with Intel optimization tools
- diffusers
- openvino
- optimization
- transformers
Ollama alternative for Rockchip NPU: An efficient solution for running AI and Deep learning models on Rockchip devices with optimized NPU support ( rkllm )
[NeurIPS 2025] The one-stop repository for LLM unlearning
- unlearning
- privacy-protection
- right-to-be-forgotten
- llm-unlearning
GPU worker client for the Talos network. Pairs with your Talos account, serves open-model inference jobs over a WebSocket, and reports uptime for payouts.
Fast, exact inference for StartLux-Decision models on NVIDIA GPUs: the released GGUF files on int8 and 4-bit tensor cores, behind the /v1/systemone API, with up to 5.3× lower latency than vLLM in FP8 and bit-identical answers.
- calibration
- cuda
- cuda-graphs
- decision-making
Local inference for diffusion language models (LLaDA, Dream), written in Mojo
- cpu
- diffusion-models
- gguf
- inference
🔥🔥 The extremely fast inference engine for decision model
- gemma
- inference-engine
- llm-inference
- llm-serving
Xolotl: run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM): ~33 tok/s on agent tasks, ~41 in fast mode, 512K context. Plus GLM-5.3-Flash, Qwen3.8 and LFM3-Omni. MoE experts tiered across VRAM, pinned RAM and NVMe, with Xolotl Direct NVMe->VRAM. OpenAI + Anthropic API.
A curated list of tools built for Jev — TypeSafe AI's System One model for typed decisions.
- jev
- awesome-list
- jev-ai
- jev-alternative
🧱 Experimental library for building inference engines
AI Agent Identity and Access Management and Inference Firewall
- agent-identity
- ai-audit
- ai-firewall
- ai-gateway
Dyno Lab — an AI safety, alignment and interpretability research workbench for Apple Silicon. Explore activations, probes, SAEs and interventions with local inference, a Python SDK, APIs and MCP.
- ai
- ai-interpretability
- ai-safety
- aisafety
llama.cpp fork for significantly improved performance on Ampere generation Nvidia cards (especially RTX 3090 / 3090 Ti): example: 110+ tok/s over a 100K-token generation at temperature 1 for qwen3.8, supporting full 262K ctx. Supports low memory (12gb) configs for 200k+ ctx. 120+ TPS, 3300+ prefill for QFN 3.8.
- ampere
- consumer-gpu
- cuda
- ggml
C++/CUDA single-GPU inference engine for Qwen3.8 (incl. Qwen3.8-Flash-Next with offloaded experts) on the RTX 5090; grew from NInfer
Berth is a lightweight GPU resource manager written in Go. It controls which applications get access to your GPU, when they get it, and how that access is handed off.
Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.