Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
Repository Index
57,138 visible repositories in the index
1–20 of 20 · Most stars
OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.
- evaluation
- benchmark
- large-language-model
- llm
Translated非线智能 NoneLinear (Non-linear Intelligence) - ReLE evaluation: capability evaluation of Chinese AI large models (continuously updated): it currently covers 374 large models, including commercial models such as chatgpt, gpt-5.4, Google gemini-3.1-pro, Claude-4.6, 文心 ERNIE-X1.1, ERNIE-5.0, qwen3.6-max, qwen3.6-plus, Baichuan, iFlytek Spark, and SenseTime senseChat, as well as open-source large models such as step3.5-flash, kimi-k2.6, ernie4.5, MiniMax-M2.7, deepseek-v4, Qwen3.6, llama4, Zhipu GLM-5.1, MiMo-V2, LongCat, gemma4, and mistral. It provides not only leaderboards but also a large model defect library with more than 2,000,000 entries! This makes it convenient for the broad community to research, analyze, and improve large models.
- agentic-ai
- artificial-intelligence
- llm-agent
- llm-evaluation
SWE-bench: Can Language Models Resolve Real-world Github Issues?
- benchmark
- language-model
- software-engineering
Benchmark for tracking model capability after release.
A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBench.
This repository introduces PIXIU, an open-source resource featuring the first financial large language models (LLMs), instruction tuning data, and evaluation benchmarks to holistically assess financial LLMs. Our goal is to continually push forward the open-source development of financial artificial intelligence (AI).
- aifinance
- fintech
- llama
- machine-learning
TranslatedSprix AI at 屿智同行 (Yuzhi Tongxing, "island wisdom traveling together") × 保研岛 (Baoyan Dao, "Baoyan Island") — an open-source benchmark for AI agents in Baoyan advising and application planning.
🔥 The first-ever interactive streaming video generation benchmark.
- avatar-performance
- interaction-streaming-video
- real-time-video-editing
- streaming-video-generation
Aixio Layer v2 recovers editable layers from flat designs where possible. 150-design benchmark vs Canva Magic Layers, Qwen-Image-Layer and GPT-6 Astra Ultra.
- ai-design
- benchmark
- image-to-layers
- layer-decomposition
A lightweight Python SDK and CLI for evaluating recursive self-improvement (RSI) methods across agent benchmarks.
- ai-agents
- benchmarking
- llm-evaluation
- python
:sunrise: A comprehensive, cutting-edge open-source framework for next-generation Transformers/LLM, with compute-matched benchmarking and a platform for autonomous architecture research.
- auto-research
- benchmark
- deep-learning
- language-models
TavernGym and TavernBench: a faithful, full-content Hearthstone Battlegrounds simulator with an RL interface, and a 960-task decision benchmark. Code and data for "Build, Don't Play".
- game-ai
- hearthstone
- reinforcement-learning
- rust
The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.
- ai-agents
- benchmark
- coding-agents
- embodied-ai
Can AI agents rebuild websites through browser exploration?
- benchmark
- harbor
- website-reconstruction
- agent-evaluation
How many bugs can LMs find & fix in large codebases?
Enterprise-grade benchmark evaluating whether AI agents deliver verifiable business outcomes in real business systems — general domain, finance, supply chain, and manufacturing.
- mfg
- scm
- erp
- eval
A Long-Horizon Benchmark for Self-Improving Inference Systems
DecisionPivot: a diagnostic benchmark for decision models in LLM systems. Contrast pairs that differ in one decisive detail, generators with independent solvers, scorer and runners.
- benchmark
- calibration
- decision-making
- evaluation
Evolved DSH plugin harnesses for five agent benchmarks, with versioned artifacts, evolution curves and Life-Harness comparisons
- agent-benchmark
- dsh
- harness
- leaderboard