Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
Repository Index
111,821 visible repositories in the index
1–24 of 42 · Most stars
Generates original ARC-AGI-1-style tasks distribution-matched to the public eval set.
Open-source AI red teaming toolkit from Microsoft. Test generative AI systems with automated scans, adaptive multi-turn attacks, and a web UI.
- ai-red-team
- generative-ai
- red-team-tools
- responsible-ai
One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks
- agi
- evaluation
- large-language-models
- multimodal
Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks
- gpt-4v
- large-language-models
- llava
- multi-modal
Evaluation and Tracking for LLM Experiments and AI Agents
- machine-learning
- neural-networks
- explainable-ml
- llmops
LLM-as-a-Verifier is a general-purpose framework that provides fine-grained feedback for any agent without requiring additional training. It achieves SOTA performance across coding, robotics, and medical agentic benchmarks.
[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
- agent
- artificial-intelligence
- benchmark
- multimodal
Repository for benchmarking graph neural networks (JMLR 2023)
- graph-representation-learning
- graph-neural-networks
- benchmark-framework
- graph-deep-learning
[NeurIPS 2026] Safety failures in coding agents and frontier LLMs. Reproducible workspaces for studying when AI agents generate unsafe data during task completion.
A benchmark for LLMs on complicated tasks in the terminal
UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.
- machine-learning
- experimentation
- llm-prompting
- llm-test
👾 nao is the #1 open source agentic BI. (1) Create context with nao-core cli, (2) deploy nao chat interface for everyone
- agentic-analytics
- analytics
- analytics-engineering
- bigquery
Measuring frontier coding agents on original, long-horizon engineering tasks
The WeightWatcher tool for predicting the accuracy of Deep Neural Networks
Measuring Massive Multitask Language Understanding | ICLR 2021
- muti-task
- transfer-learning
- gpt-3
- few-shot-learning
A powerful tool for automated LLM fuzzing. It is designed to help developers and security researchers identify and mitigate potential jailbreaks in their LLM APIs.
Advbox is a toolbox to generate adversarial examples that fool neural networks in PaddlePaddle、PyTorch、Caffe2、MxNet、Keras、TensorFlow and Advbox can benchmark the robustness of machine learning models. Advbox give a command line tool to generate adversarial examples with Zero-Coding.
- adversarial-examples
- paddlepaddle
- machine-learning
- security
The MATH Dataset (NeurIPS 2021)
- math
- reasoning
- logic
Build, Evaluate, and Deploy GUI Agents — online RL training, standardized benchmarks, and real-device deployment in one framework.
- agentrl
- guiagents
- onlinerl
- openclaw
A Toolbox for Adversarial Robustness Research
- pytorch
- adversarial-examples
- adversarial-example
- adversarial-perturbations
Evaluate and improve models and agents using environments
- reinforcement-learning
- reinforcement-learning-environments
- rl-environment
- rl-training
Open-source benchmark for browser AI agents on daily tasks.
This repository allows reproduction of Poetiq's record-breaking submission to the ARC-AGI-1 and ARC-AGI-2 benchmarks.