LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.
Repository Index
132,241 visible repositories in the index
1–6 of 6 · Most stars
Run 284B DeepSeek-V4-Flash (<59GiB RAM: 41.1 cold pp, 9.1 N=1 tg, up to 25.3 spec tok/s) & 30B MoEs (22.4 N=1 / 111.6 spec tok/s) on 64GB CPU (0 GPUs).
Pure Rust Inference Engine
- cuda
- gb10
- llm-inference
- mamba
llama.cpp fork for significantly improved performance on Ampere generation Nvidia cards (especially RTX 3090 / 3090 Ti): example: 110+ tok/s over a 100K-token generation at temperature 1 for qwen3.8, supporting full 262K ctx. Supports low memory (12gb) configs for 200k+ ctx. 120+ TPS, 3300+ prefill for QFN 3.8.
- ampere
- consumer-gpu
- cuda
- ggml
ChatGPT cross-platform client made with the Quasar Framework
Ideas, tricks, materials with puzzles for AIs and some wetware