An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
Repository Index
47,133 repositories indexed
1–24 of 30 · Most stars
🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
- apple
- cua
- lume
- macos
The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.
- machine-learning
- ai
- ml
- mlflow
Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
The LLM Evaluation Framework
- evaluation-metrics
- evaluation-framework
- llm-evaluation
- llm-evaluation-framework
Supercharge Your LLM Application Evaluations 🚀
- llm
- llmops
- evaluation
Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced analytics with timeline and execution graph view
- agentneo
- agents
- ai-performance-optimization
- llm-testing
A framework for few-shot evaluation of language models.
- evaluation-framework
- language-model
- transformer
Gorilla: Training and Evaluating LLMs for Function Calls (Tool Calls)
AI Observability & Evaluation
- llmops
- ai-monitoring
- ai-observability
- llm-eval
the LLM vulnerability scanner
- ai
- llm-evaluation
- llm-security
- security-scanners
Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
- data-drift
- jupyter-notebook
- pandas-dataframe
- machine-learning
Open-Source Agent Observability & Evals for AI Platform teams. Trace, test, route and govern every agent & coding assistant in the company.
TranslatedOpen-source AI API relay benchmark, rankings & switcher for Claude Code / Codex: measured availability, latency and price
Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.
- ai-gateway
- observability
- simulation
- ai-agents
Evaluation code for various unsupervised automated metrics for Natural Language Generation.
- natural-language-generation
- natural-language-processing
- nlg
- nlp
Trace-native CI/CD for AI agents: grade production traces, cluster failures, freeze them into replayable regression tests, and block PRs that regress. Self-hostable.
A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBench.
FactCircuit: an auditable claim-and-evidence verification loop with immutable provenance, staged checks, bounded retrieval, and reproducible evaluation.
- accuracy-decline
- ai-safety
- audit-trail
- fact-checking
This repository introduces PIXIU, an open-source resource featuring the first financial large language models (LLMs), instruction tuning data, and evaluation benchmarks to holistically assess financial LLMs. Our goal is to continually push forward the open-source development of financial artificial intelligence (AI).
- aifinance
- fintech
- llama
- machine-learning
A unified framework for building, evaluating, and recursively improving embodied agents. 🚀 Code release coming soon.