LangChain4j is an idiomatic, open-source Java library for building LLM-powered applications on the JVM. It offers a unified API over popular LLM providers and vector stores, and makes implementing tool calling (including MCP support), agents and RAG easy. It integrates seamlessly with enterprise Java frameworks like Quarkus and Spring Boot.
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
High-speed Large Language Model Serving for Local Deployment
Open-source inference platform for language, vision, embedding, reranking, image, audio, and video models. Deploy with multiple engines across heterogeneous hardware, from a laptop to a distributed cluster. Serve models through OpenAI-compatible APIs.
Easily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!
A Datacenter Scale Distributed Inference Serving Framework
An Extensible Toolkit for Finetuning and Inference of Large Foundation Models. Large Models for All.
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
TranslatedOpen-source manuscript of 《深入理解 AI Infra:量化分析与系统设计》 (In-Depth Understanding of AI Infra: Quantitative Analysis and System Design), by 李博杰 (Li Bojie): starting from hardware constraints and model architecture, it quantitatively derives the system design of LLM inference and training. Includes the full text of the book, a PDF, and companion calculation tools and experiments
- accelerator
- ai-infra
- ai-infrastructure
- book
Open source inference engine that tunes its kernels on your hardware to find the fastest way to run each model. 2x faster than Ollama. Works on Apple Silicon, NVIDIA, AMD, Intel, or just a CPU.
Fast, flexible LLM inference
Infrastructure for continually self‑improving agents
TranslatedAI video note generation tool: let AI take notes for your videos
Home of StarCoder: fine-tuning & inference!
Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook
Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay focused on your agents core logic.
lightweight, standalone C++ inference engine for Google's Gemma models.
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
[EMNLP'23, ACL'24] To speed up LLMs' inference and enhance LLM's perceive of key information, compress the prompt and KV-Cache, which achieves up to 20x compression with minimal performance loss.
Modeling, training, eval, and inference code for OLMo
Code for the paper "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer"
An open, programmable decision layer for models and compute.
- mixture-of-models
- semantic-router
- vllm
- ai-gateway