Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.
Repository Index
107,098 visible repositories in the index
217–238 of 238 · Most stars
Kubernetes operator for self-hosted LLM inference across a heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. Runtimes: llama.cpp, vLLM, TGI, mlx-server. Multi-GPU sharding, model caching, OpenAI-compatible endpoints. Apache-2.0, run across homelab and on-prem fleets, actively developed.
Fast local LLM inference for the GPU you already have: one Rust binary, GGUF models, an OpenAI-compatible server and a chat page with a ghost.
- cuda
- desktop-app
- gguf
- llm
Reproducible Docker/vLLM deployment, tuning, and benchmarks for DeepSeek V4.1 Flash on a switchless three-node or four-node DGX Spark fabric.
llama.cpp fork that abliterates at inference
TranslatedChinese edition of an AI reference image gallery | Costumes (1,407 realistic sets / 1,320 animation sets), 1,499 face shapes, 1,439 pose line drawings and 1,439 depth maps, 843 anime-style (二次元) entries. Designed for AI painting, character design, and comic-drama storyboards, with support for Chinese search, multi-angle browsing, and image copying and downloading. Open-source static site that preserves the original sources and terms of use.
- ai-art
- ai-comics
- art-reference
- character-design
Deepinfer inspired implemetation on SM80 Ampere architecture. Deepest respect for DeepSeek opensource this work!
ModelSphere is an open-source LLM inference infrastructure designed to make production-grade model serving simple, efficient, and continuously optimized.
PROMETHEVS: run the full DeepSeek-V4.1-Flash-1017 (510 GB) on a gaming PC with 64 GB RAM and a 12 GB GPU. PROMETHEVS engine (compress-then-correct), OpenAI-compatible local server, report and logs.
Production compatibility gateway bridge & observability platform connecting Claude Code CLI with Google Antigravity & Gemini inference models via Anthropic Messages API
PocketStrike is a privacy-first mobile AI workstation. It combines zero-cloud native on-device GGUF inference (ARM NEON accelerated) │ with frontier cloud LLMs, multi-step ReAct agent tools, Model Context Protocol (MCP) servers, self-evolving cross-session memory, and FLUX neural image │ generation.
Source code of Local Code, a private AI agent for Mac, iPad and iPhone. Runs DeepSeek 4.1 on device, with Python, Node.js and a shell built into the app.
A Long-Horizon Benchmark for Self-Improving Inference Systems
DecisionPivot: a diagnostic benchmark for decision models in LLM systems. Contrast pairs that differ in one decisive detail, generators with independent solvers, scorer and runners.
- benchmark
- calibration
- decision-making
- evaluation
TranslatedAn AI business workbench for small and medium-sized auto parts manufacturers: real product catalog, prospect discovery and evidence review, RAG-based document Q&A, multi-turn inquiries, human handoff and quotation follow-up, with reproducible tiered evaluation.
Curated Gemini Omni prompts, API guide, and video examples — video generation, editing, camera control, style transfer, text rendering, and multi-modal workflows.
- ai-prompts
- deepmind
- gemini
- gemini-omni
ReAlign models instantly, on demand. Per-request alignment for verified users and channels, or Always On. Use directly or with Hermes, OpenClaw and custom harnesses.
Modern desktop application (Rust + Tauri v2 + Svelte 5 + Candle (HF)) for communicating with AI models that runs completely locally on your computer. No subscriptions, no data sent to the internet — just you and your personal AI assistant
- ai-chat
- artificial-intelligence
- candle
- chatbot
Inference Llama in Common Lisp
🚀 LLM inference Engine in Swift/Metal, Load GGUF and safe tensors modes, no conversion, no cpp, pure swift
TranslatedA Chinese-language guide to digital subscriptions and AI tools, including subscription instructions, risk warnings, and an FAQ.
gtfs2vec - Learning GTFS Embeddings for comparing PublicTransport Offer in Microregions