Repository Index

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

• apple-silicon
• inference-server
• llm
• macos

Run 35B MoE models in ~2.5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported

Run a 105 GB AI model on a Mac that can't hold it. Qwen3.8-Flash-Next 4-bit, streamed from SSD on 16–64 GB Apple Silicon Macs. Fully native Swift, MLX and Metal. Offline, no Python. CLI, Swift library, and Ollama/OpenAI/Anthropic APIs. Works with Claude Code and Codex.

• apple-silicon
• llm
• llm-inference
• local-llm

Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.

• apple-silicon
• cuda
• gguf
• inference