Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
Repository Index
47,133 repositories indexed
1–19 of 19 · Most stars
Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
The LLM Evaluation Framework
- evaluation-metrics
- evaluation-framework
- llm-evaluation
- llm-evaluation-framework
Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced analytics with timeline and execution graph view
- agentneo
- agents
- ai-performance-optimization
- llm-testing
A framework for few-shot evaluation of language models.
- evaluation-framework
- language-model
- transformer
Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
- data-drift
- jupyter-notebook
- pandas-dataframe
- machine-learning
Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.
- ai-gateway
- observability
- simulation
- ai-agents
A unified framework for building, evaluating, and recursively improving embodied agents. 🚀 Code release coming soon.
Scikit-learn for Recursive Self-improvement (RSI). A modular framework for recursive harness evolution.
- meta-learning
- reinforcement-learning
- rsi
- ai-agents
A Unified Framework for Efficient RSI Research
- agent
- recursive-self-improvement
:sunrise: A comprehensive, cutting-edge open-source framework for next-generation Transformers/LLM, with compute-matched benchmarking and a platform for autonomous architecture research.
- auto-research
- benchmark
- deep-learning
- language-models
Feather agent harness, training integrations, and evaluation tools for knowledge work
How many bugs can LMs find & fix in large codebases?
DiscoveryRSI: Recursively Learning to Discover Valuable Hypotheses
- agent-framework
- agentic-workflow
- aiforscience
- llm-agents
LLM and agent evaluation for Java & Kotlin. Runs in JUnit and CI. Spring AI, LangChain4j, Koog, Embabel, and any LLM client.
- agent-evaluation
- evaluation-framework
- evaluation-metrics
- java
The E2E AI testing tool | No ML Overhead
Framework-agnostic evaluation harness for Go — test your MCP servers and AI agents with scored, CI-ready checks.
Catch silent failures in agent workflows by verifying real system state