🪢 Open source AI engineering platform: AI Gateway, agent observability, evals, and prompt/skill management, built on OpenTelemetry. One open platform for every AI call and every agent. Self-host or use Langfuse Cloud.
Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
The LLM Evaluation Framework
- evaluation-metrics
- evaluation-framework
- llm-evaluation
- llm-evaluation-framework
Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
Train the smallest LM you can that fits in 16MB. Best model wins!
Set of tools to assess and improve LLM security.
Comet: agent skill harness for turning ideas into evaluated workflows
Same agent, Smaller bill. The tools and infrastructure to get more work from your AI. Fewer tokens. Less compute. Lower cost.
mean Average Precision - This code evaluates the performance of your neural net for object recognition.
Formal verification for AI: a Lean 4 + Dafny proof checker that catches fake AI proofs (reward hacking), and verified recursive self-improvement (RSI) for coding agents. AI for math, physics and software engineering.
Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
- evaluation
- evaluation-framework
- evaluation-metrics
- huggingface
Machine learning metrics for distributed, scalable PyTorch applications.
AssetOpsBench - Industry 4.0: A unified benchmark and framework for building, orchestrating, and evaluating domain-specific AI agents for Industry 4.0 asset operations and maintenance, with 460+ scenarios, 5 specialist agents (IoT, FMSR, TSFM, Work Order,...), and multi-agent orchestration blueprints (MetaAgent, AgentHive) over MCP.
The Fable Workflow: how Claude Fable 5 worked, distilled into skills any model can run, with the eval that keeps it honest. Think / act / prove.
Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.
Evaluation code for various unsupervised automated metrics for Natural Language Generation.
Trace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.
HOTA (and other) evaluation metrics for Multi-Object Tracking (MOT).
BLEURT is a metric for Natural Language Generation based on transfer learning.
A lightweight Python SDK and CLI for evaluating recursive self-improvement (RSI) methods across agent benchmarks.
:dog: The open-source Gauge CLI. Hillclimb your Skills, CLI, SDK, MCP, API, or Docs to make something agents want.
LLM and agent evaluation for Java & Kotlin. Runs in JUnit and CI. Spring AI, LangChain4j, Koog, Embabel, and any LLM client.
- agent-evaluation
- evaluation-framework
- evaluation-metrics
- java