A high-throughput and memory-efficient inference and serving engine for LLMs
Repository Index
132,241 visible repositories in the index
1–24 of 80 · Most stars
Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)
- fine-tuning
- llama
- llm
- peft
A hierarchical inference engine for frontier MoE models: hot experts on the GPU (CUDA, Vulkan), warm ones in RAM, cold ones streamed from NVMe. Written in C, runs on the hardware you already own.
💖🧸 Self hosted, you-owned Grok Companion, a container of souls of waifu, cyber livings to bring them into our worlds, wishing to achieve Neuro-sama's altitude. Capable of realtime voice chat, Minecraft, Factorio playing. Web / macOS / Windows supported.
- ai-vtuber
- live2d
- neuro-sama
- neurosama
SGLang is a high-performance serving framework for large language models and multimodal models.
How Python does AI. Agents, realtime voice, image generation, embeddings. Every model, every interface, typed end to end.
- agent-framework
- pydantic
- python
- genai
Instruct-tune LLaMA on consumer hardware
FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
- deepseek-v4
- edge
- glm
- inference
Use PEFT or Full-parameter to CPT/SFT/DPO/GRPO 600+ LLMs (Qwen3.8, DeepSeek-V4, GLM-5.1, InternLM3, Llama4, ...) and 300+ MLLMs (Qwen3-VL, Qwen3-Omni, InternVL3.5, Ovis2.5, GLM5.3, Gemma4, Llava, Phi4, ...) (AAAI 2025).
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.
A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.
- avx2
- c99
- cpu-inference
- deep-learning
AI Observability & Evaluation
- llmops
- ai-monitoring
- ai-observability
- llm-eval
Device-native foundation models: 145B on a laptop, 35B on a phone. Latent MoE, SSD-streamed weights, a router that predicts loads a step ahead. Mac, iPhone, Android, Windows.
- edge-ai
- foundation-model
- on-device-ai
🚀🧠💬 Supercharged Custom Instructions for ChatGPT (non-coding) and ChatGPT Advanced Data Analysis (coding).
A Next-Generation Training Engine Built for Ultra-Large MoE Models
Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning
GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models
TranslatedEmbracing All Creation (Desktop) — Dedicated to the Harmonious Coexistence of Hundreds of DSH Plugins
把你在各处注册的 New API / One API / OneHub / DoneHub / Veloera / AnyRouter / Sub2API 等站点, 汇聚成 一个 API Key、一个入口,自动发现模型、智能路由、成本最优
Run GLM-5.3 (744B), DeepSeek-V4.1-Flash (748B), MiniMax-M3.1-Flash (428B), GLM-5.3-Flash (321B) and Hy4-Flash (220B) on your own GPUs - from 8 GB of VRAM, NVIDIA and AMD, one dashboard, OpenAI- and Anthropic-compatible API.
- cuda
- glm
- llm
- llm-inference
We unified the interfaces of instruction-tuning data (e.g., CoT data), multiple LLMs and parameter-efficient methods (e.g., lora, p-tuning) together for easy use. We welcome open-source enthusiasts to initiate any meaningful PR on this repo and integrate as many LLM related technologies as possible. 我们打造了方便研究人员上手和使用大模型等微调平台,我们欢迎开源爱好者发起任何有意义的pr!
UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.
- machine-learning
- experimentation
- llm-prompting
- llm-test
A fully functional AI Agent desktop client that supports Webui access and can be creatively customized and expanded!
- ai-agents
- ai-tools
- claude-code
- claude-desktop
Aila(AI超元域): The premier AI integration tool for Windows, macOS, and Android. Ask once, get answers from 10+ AIs like ChatGPT, Gemini, Claude3, Copilot, Poe, perplexity and more. Features customizable AI and prompts.
- autogpt
- openai
- openai-api
- chatgpt