LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
Repository Index
142,273 visible repositories in the index
1–22 of 22 · Most stars
A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations
Sub-50mb agentic, voice and vision intelligence for consumer devices.
- cactus
- llm
- on-device-ai
- edge-ai
RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.
- attention-mechanism
- deep-learning
- gpt
- gpt-2
TranslatedOpen-source manuscript of 《深入理解 AI Infra:量化分析与系统设计》 (In-Depth Understanding of AI Infra: Quantitative Analysis and System Design), by 李博杰 (Li Bojie): starting from hardware constraints and model architecture, it quantitatively derives the system design of LLM inference and training. Includes the full text of the book, a PDF, and companion calculation tools and experiments
- accelerator
- ai-infra
- ai-infrastructure
- book
[EMNLP'23, ACL'24] To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.
See what Claude Code and Codex actually send to the API — and what each part costs.
- claude-code
- codex-cli
- cost-tracking
- observability
Open-source multimodal retrieval engine (Morphik Core)
- artificial-intelligence
- cache-augmented-generation
- colpali
- database
Cache-Augmented Generation: A Simple, Efficient Alternative to RAG
Zero-GPU CPU inference for 30B-284B MoE models (Qwen3.8 176.5B cold-cache certified in 28.26/48.35 GiB Cgroup v2, DeepSeek-V4 284B, Nemotron-30B) + RouteCrack 2.0.
A LLM semantic caching system aiming to enhance user experience by reducing response time via cached query-result pairs.
- llm
- semantic-cache
vMLX - Use MLX models easily - JANGQ (GGUF for MLX) - Not dependant on mlx_vlm
- llm
- lmstudio
- macbook
- mlx
LLM notes, including model inference, transformer model structure, and llm framework code analysis notes.
- llm
- llm-inference
- triton-kernels
- vllm
The online version is temporarily unavailable because we cannot afford the key. You can clone and run it locally. Note: we set defaul openai key. If keys exceed plan and are invalid, please tell us. The response speed depends on openai. ( sometimes, the official is too crowded and slow)
- chatgpt
- information-extraction
- zero-shot
- event-extraction
Frontier-class open models on a free Kaggle TPU v5e-8: GLM-5.3-Flash 320B MoE (~165 tok/s, our own JAX engine) and Qwen3.8-27B bf16 (~130 tok/s), 262k context, prefix caching. Works with Claude Code, Codex, opencode and pi.
- claude-code
- coding-agents
- free-gpu
- jax
[ICLR'26] The official code implementation for "Cache-to-Cache: Direct Semantic Communication Between Large Language Models"
- kv-cache
- llm
- multi-agent
Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation (NeurIPS 2025)
- adaptive-computation
- early-exiting
- kv-cache
- llm
Self-hosted AI agent OS. Your memory, chat, agents, and files stay on hardware you own, offline by default, cloud by choice. Offline AI memory (taOSmd), self-hosted multi-framework group chat, a full web desktop + app store, and auto-clustering across the consumer hardware you already have (Orange/Raspberry Pi, Mac mini, gaming PC).
- agent-framework
- ai-agents
- ai-platform
- apple-silicon
Use ArXiv ChatGuru to talk to research papers. This app uses LangChain, OpenAI, Streamlit, and Redis as a vector database/semantic cache.
- ai
- machine-learning
- openai
- python
From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.
- cuda
- flash-attention
- fsdp
- inference-engine
🔥🔥 The extremely fast inference engine for decision model
- gemma
- inference-engine
- llm-inference
- llm-serving
Official implementation of paper "Cache Engineering for Agents"