🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
Repository Index
47,133 repositories indexed
1–16 of 16 · Most stars
The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.
- machine-learning
- ai
- ml
- mlflow
Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced analytics with timeline and execution graph view
- agentneo
- agents
- ai-performance-optimization
- llm-testing
Build high-quality LLM apps - from prototyping, testing to production deployment and monitoring.
Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.
- ai-gateway
- observability
- simulation
- ai-agents
Trace-native CI/CD for AI agents: grade production traces, cluster failures, freeze them into replayable regression tests, and block PRs that regress. Self-hostable.
This repository introduces PIXIU, an open-source resource featuring the first financial large language models (LLMs), instruction tuning data, and evaluation benchmarks to holistically assess financial LLMs. Our goal is to continually push forward the open-source development of financial artificial intelligence (AI).
- aifinance
- fintech
- llama
- machine-learning
Build AI phone agents on your own computer and test every change with an AI caller. Open source, runs free with no API keys, cents per call with your own keys. Imports Dialogflow CX.
- agents-as-code
- contact-center
- conversational-ai
- cx-agent-studio
Build AI text detectors whose decisions are mechanistically grounded and human-auditable (EMNLP 2026)
- ai-text-detection
- explainable-ai
- interpretability
- mechanistic-interpretability
A lightweight Python SDK and CLI for evaluating recursive self-improvement (RSI) methods across agent benchmarks.
Aixio Layer v2 recovers editable layers from flat designs where possible. 150-design benchmark vs Canva Magic Layers, Qwen-Image-Layer and GPT-6 Astra Ultra.
- ai-design
- benchmark
- image-to-layers
- layer-decomposition
Official Repository of paper OmniReasoning: Pushing the Limits of Audio-Visual Joint Reasoning
DecisionPivot: a diagnostic benchmark for decision models in LLM systems. Contrast pairs that differ in one decisive detail, generators with independent solvers, scorer and runners.
- benchmark
- calibration
- decision-making
- evaluation
Fault-injected tool environments with side-effect grading for evaluating and post-training LLM agents
LLM and agent evaluation for Java & Kotlin. Runs in JUnit and CI. Spring AI, LangChain4j, Koog, Embabel, and any LLM client.
- agent-evaluation
- evaluation-framework
- evaluation-metrics
- java
Catch your AI agents when they lie about what they shipped — verifies claims against git instead of believing the agent.
- agent-orchestration
- cli
- developer-tools
- llm-agents