An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
Repository Index
109,091 visible repositories in the index
1–24 of 106 · Most stars
🪢 Open source AI engineering platform: AI Gateway, agent observability, evals, and prompt/skill management, built on OpenTelemetry. One open platform for every AI call and every agent. Self-host or use Langfuse Cloud.
- analytics
- llm
- llmops
- large-language-models
The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.
- machine-learning
- ai
- ml
- mlflow
Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
The LLM Evaluation Framework
- evaluation-metrics
- evaluation-framework
- llm-evaluation
- llm-evaluation-framework
Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced analytics with timeline and execution graph view
- agentneo
- agents
- ai-performance-optimization
- llm-testing
A framework for few-shot evaluation of language models.
- evaluation-framework
- language-model
- transformer
Gorilla: Training and Evaluating LLMs for Function Calls (Tool Calls)
TranslatedA Codex jailbreak prompt and test pack for gpt.
Generates original ARC-AGI-1-style tasks distribution-matched to the public eval set.
CodeGeeX: An Open Multilingual Code Generation Model (KDD 2023)
- code-generation
- pretrained-models
- tools
TranslatedBELLE: Be Everyone's Large Language model Engine (open-source Chinese conversational large language model)
- bloom
- instruction-set
- llama
- open-models
Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
- data-drift
- jupyter-notebook
- pandas-dataframe
- machine-learning
GLM-130B: An Open Bilingual Pre-Trained Model (ICLR 2023)
This repo powers my experiment where ChatGPT manages a real-money micro-cap stock portfolio.
Translated非线智能 NoneLinear (Non-linear Intelligence) - ReLE evaluation: capability evaluation of Chinese AI large models (continuously updated): it currently covers 374 large models, including commercial models such as chatgpt, gpt-5.4, Google gemini-3.1-pro, Claude-4.6, 文心 ERNIE-X1.1, ERNIE-5.0, qwen3.6-max, qwen3.6-plus, Baichuan, iFlytek Spark, and SenseTime senseChat, as well as open-source large models such as step3.5-flash, kimi-k2.6, ernie4.5, MiniMax-M2.7, deepseek-v4, Qwen3.6, llama4, Zhipu GLM-5.1, MiMo-V2, LongCat, gemma4, and mistral. It provides not only leaderboards but also a large model defect library with more than 2,000,000 entries! This makes it convenient for the broad community to research, analyze, and improve large models.
- agentic-ai
- artificial-intelligence
- llm-agent
- llm-evaluation
Agent OS: the agent gets smarter on its own. We just hold the line: Interview-gated, staged evaluation, budgeted evolution loop. MCP server, 14 runtimes: Claude Code, Codex CLI, Gemini CLI, OpenCode, Copilot, Kiro and more.
- ai-agent
- agent-os
- llm-orchestration
- agentic-ai
🐢 Open-Source Evaluation & Testing library for LLM Agents
- mlops
- ml-validation
- ml-testing
- ai-testing
Democratizing Reinforcement Learning for LLMs
- agent-framework
- agentic-workflow
- distributed-training
- llm-training
A visual playground for agentic workflows: Iterate over your agents 10x faster
[ICLR'24 spotlight] An open platform for training, serving, and evaluating large language model for tool learning.
🧩 FrontierAgent, our agent framework, open-sourced alongside it — native command-line TUI, ReAct and Agent Team modes, one command on macOS and Linux, no preinstall, no hard Docker dependency.
- agent-orchestration
- ai-agents
- harness
- agentic-ai
Train the smallest LM you can that fits in 16MB. Best model wins!