Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
Repository Index
47,133 repositories indexed
1–19 of 19 · Most stars
Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
The LLM Evaluation Framework
- evaluation-metrics
- evaluation-framework
- llm-evaluation
- llm-evaluation-framework
Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced analytics with timeline and execution graph view
- agentneo
- agents
- ai-performance-optimization
- llm-testing
A framework for few-shot evaluation of language models.
- evaluation-framework
- language-model
- transformer
Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
- data-drift
- jupyter-notebook
- pandas-dataframe
- machine-learning
Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.
- ai-gateway
- observability
- simulation
- ai-agents
Steering-first neuron explorer for Qwen3.5-4B: push every MLP neuron, keep every rollout, browse the results
- interpretability
- mechanistic-interpretability
- qwen
- steering
A unified framework for building, evaluating, and recursively improving embodied agents. 🚀 Code release coming soon.
Scikit-learn for Recursive Self-improvement (RSI). A modular framework for recursive harness evolution.
- meta-learning
- reinforcement-learning
- rsi
- ai-agents
An agent that compiles git maintenance chores into reusable programs, then decides whether to replay them by running checks rather than by guessing the task looks familiar
- agent-architecture
- git-automation
- llm-agents
- program-synthesis
A Unified Framework for Efficient RSI Research
- agent
- recursive-self-improvement
:sunrise: A comprehensive, cutting-edge open-source framework for next-generation Transformers/LLM, with compute-matched benchmarking and a platform for autonomous architecture research.
- auto-research
- benchmark
- deep-learning
- language-models
The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.
- ai-agents
- benchmark
- coding-agents
- embodied-ai
How many bugs can LMs find & fix in large codebases?
DiscoveryRSI: Recursively Learning to Discover Valuable Hypotheses
- agent-framework
- agentic-workflow
- aiforscience
- llm-agents
LLM and agent evaluation for Java & Kotlin. Runs in JUnit and CI. Spring AI, LangChain4j, Koog, Embabel, and any LLM client.
- agent-evaluation
- evaluation-framework
- evaluation-metrics
- java
The E2E AI testing tool | No ML Overhead
Framework-agnostic evaluation harness for Go — test your MCP servers and AI agents with scored, CI-ready checks.