🔥🔥 The extremely fast inference engine for decision model
- gemma
- inference-engine
- llm-inference
- llm-serving
Repository Index
56,638 visible repositories in the index
193–216 of 245 · Most stars
Fast, exact inference for StartLux-Decision models on NVIDIA GPUs: the released GGUF files on int8 and 4-bit tensor cores, behind the /v1/systemone API, with up to 5.3× lower latency than vLLM in FP8 and bit-identical answers.
- calibration
- cuda
- cuda-graphs
- decision-making
AI Agent Identity and Access Management and Inference Firewall
- agent-identity
- ai-audit
- ai-firewall
- ai-gateway
🧱 Experimental library for building inference engines
Threat modeling for AI coding agents: point Claude Code, Codex or any agent at your repo and get an evidence-traced threat model. 36 architectures, 29 methods, MIT.
An agent that compiles git maintenance chores into reusable programs, then decides whether to replay them by running checks rather than by guessing the task looks familiar
- agent-architecture
- git-automation
- llm-agents
- program-synthesis
Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.
Pathology Language and Image Pre-Training (PLIP) is the first vision and language foundation model for Pathology AI (Nature Medicine). PLIP is a large-scale pre-trained model that can be used to extract visual and language features from pathology images and text description. The model is a fine-tuned version of the original CLIP model.
- artificial-intelligence
- clip
- pathology
- vision-and-language
TranslatedThe operational superset of Pi Coding Agent — everything Pi, plus observability, governance, recovery, evaluation and multi-agent orchestration.
- pi
- ai-agent
- pi-agent
- pi-coding-agent
Extract knowledge from all information sources using gpt and other language models. Index and make Q&A session with information sources.
Berth is a lightweight GPU resource manager written in Go. It controls which applications get access to your GPU, when they get it, and how that access is handed off.
C++/CUDA single-GPU inference engine for Qwen3.8 (incl. Qwen3.8-Flash-Next with offloaded experts) on the RTX 5090; grew from NInfer
Our lightning-fast inference engine.
From a universe of models to Collective Model Intelligence.
- collective-model-intelligence
- model-capabilities
- model-collaboration
Kubernetes operator for self-hosted LLM inference across a heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, and Apple Silicon Metal. Runtimes: llama.cpp, vLLM, TGI, mlx-server. Multi-GPU sharding, model caching, OpenAI-compatible endpoints. Apache-2.0, run across homelab and on-prem fleets, actively developed.
TranslatedPrompt editor for DSH with method cards, draft comparison and quote recovery. DeepSeek Harness (DSH) prompt engineering plugin. Spot omissions in the input box and add a sentence picked from 90 method cards, or rewrite against the original draft, check quotes and restore the original words.
Uncensored AI API and models — one key for Claude, GPT, Gemini, Grok, DeepSeek, GLM and Kimi, with uncensored and abliterated variants where available. OpenAI- and Anthropic-compatible. Pay per token.
- abliterated-models
- ai-gateway
- anthropic-compatible
- claude-api
Pure-Rust inference engine for Qwen GGUF models (Qwen3 to Qwen3.8, dense and MoE up to 770B+). CUDA and Vulkan (NVIDIA, AMD), hybrid CPU+GPU experts, MTP speculative decoding, OpenAI and Anthropic compatible server. Up to ~1.7x llama.cpp decode on a 12 GB RTX 3060.
Agent Speed — A lightweight, real-time token speed monitor for Codex and Claude Code. Track tokens per second (TPS), detect slowdowns, and monitor AI agent performance at a glance.
llama.cpp fork that abliterates at inference
Enterprise-grade benchmark evaluating whether AI agents deliver verifiable business outcomes in real business systems — general domain, finance, supply chain, and manufacturing.
- mfg
- scm
- erp
- eval
ModelSphere is an open-source LLM inference infrastructure designed to make production-grade model serving simple, efficient, and continuously optimized.
Local LLM API Intelligent Gateway — Open-source client + Server Open Core model, unified management of multiple LLM Provider channels, providing intelligent routing, session stickiness, cost optimization, full-chain observability, and team collaborative governance.
PocketStrike is a privacy-first mobile AI workstation. It combines zero-cloud native on-device GGUF inference (ARM NEON accelerated) │ with frontier cloud LLMs, multi-step ReAct agent tools, Model Context Protocol (MCP) servers, self-evolving cross-session memory, and FLUX neural image │ generation.