Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
Repository Index
47,133 repositories indexed
1–8 of 8 · Most stars
TranslatedOpen-source AI API relay benchmark, rankings & switcher for Claude Code / Codex: measured availability, latency and price
Serve large Qwen models fast on the GPUs you actually own. Qwen3.8-27B on a single 24 GB card with vLLM: 127 tok/s single-user (381 when the answer quotes the prompt), ~1,035 tok/s at 64 concurrent, 150k-262k context. vLLM patches, requant pipeline, benchmarks.
- kv-cache
- llm-inference
- local-llm
- quantization
This repository introduces PIXIU, an open-source resource featuring the first financial large language models (LLMs), instruction tuning data, and evaluation benchmarks to holistically assess financial LLMs. Our goal is to continually push forward the open-source development of financial artificial intelligence (AI).
- aifinance
- fintech
- llama
- machine-learning
A lightweight Python SDK and CLI for evaluating recursive self-improvement (RSI) methods across agent benchmarks.
How many bugs can LMs find & fix in large codebases?
Evolved DSH plugin harnesses for five agent benchmarks, with versioned artifacts, evolution curves and Life-Harness comparisons
- agent-benchmark
- dsh
- harness
- leaderboard
Prompt-injection firewall for LLM applications — 33 input detectors, 9 output scanners, federated ed25519-signed threat-intel feed. Apache 2.0, 1040 tests, F1 96.0% with 0% false positives. Docker, GitHub Action, LangChain/LlamaIndex/CrewAI integrations.
- agentic-ai
- ai-safety
- fastapi
- langchain