Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
Repository Index
57,138 visible repositories in the index
1–21 of 21 · Most stars
OpenCompass is an LLM evaluation platform, supporting a wide range of models from OpenAI, Anthropic, Gemini, Qwen, GLM, DeepSeek, etc, across 100+ datasets covering knowledge, reasoning, coding, science, language, long-context, and safety.
- evaluation
- benchmark
- large-language-model
- llm
Find the local LLM that actually runs and performs best on your hardware. Ranked by real, recency-aware benchmarks, not parameter count. One command, run it instantly.
- localllm
SWE-bench: Can Language Models Resolve Real-world Github Issues?
- benchmark
- language-model
- software-engineering
Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI
[NeurIPS 2026] DataFlex: A Unified Benchmark and Evaluation Platform for Data-Centric Training of Large Language Models
- data-mixture
- data-science
- data-selection
- data-reweighting
Benchmark for tracking model capability after release.
A deep learning toolkit for healthcare AI: EHR (MIMIC, eICU, OMOP), clinical text, signals and imaging. 30+ models, reproducible pipelines.
- healthcare
- data-mining
- deep-learning
- preprocessing
A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBench.
Tasks Assessing Protein Embeddings (TAPE), a set of five biologically relevant semi-supervised learning tasks spread across different domains of protein biology.
- deep-learning
- protein-sequences
- protein-structure
- semi-supervised-learning
TranslatedSprix AI at 屿智同行 (Yuzhi Tongxing, "island wisdom traveling together") × 保研岛 (Baoyan Dao, "Baoyan Island") — an open-source benchmark for AI agents in Baoyan advising and application planning.
🔥 The first-ever interactive streaming video generation benchmark.
- avatar-performance
- interaction-streaming-video
- real-time-video-editing
- streaming-video-generation
A lightweight Python SDK and CLI for evaluating recursive self-improvement (RSI) methods across agent benchmarks.
- ai-agents
- benchmarking
- llm-evaluation
- python
:sunrise: A comprehensive, cutting-edge open-source framework for next-generation Transformers/LLM, with compute-matched benchmarking and a platform for autonomous architecture research.
- auto-research
- benchmark
- deep-learning
- language-models
The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.
- ai-agents
- benchmark
- coding-agents
- embodied-ai
Can AI agents rebuild websites through browser exploration?
- benchmark
- harbor
- website-reconstruction
- agent-evaluation
How many bugs can LMs find & fix in large codebases?
Enterprise-grade benchmark evaluating whether AI agents deliver verifiable business outcomes in real business systems — general domain, finance, supply chain, and manufacturing.
- mfg
- scm
- erp
- eval
A Long-Horizon Benchmark for Self-Improving Inference Systems
DecisionPivot: a diagnostic benchmark for decision models in LLM systems. Contrast pairs that differ in one decisive detail, generators with independent solvers, scorer and runners.
- benchmark
- calibration
- decision-making
- evaluation
Prompt-injection firewall for LLM applications — 33 input detectors, 9 output scanners, federated ed25519-signed threat-intel feed. Apache 2.0, 1040 tests, F1 96.0% with 0% false positives. Docker, GitHub Action, LangChain/LlamaIndex/CrewAI integrations.
- agentic-ai
- ai-safety
- fastapi
- langchain