Benchmark pitfalls: how scores lie, and how to read them

Every benchmark number is a measurement taken with an imperfect instrument, on a test that may have leaked, by a system that may have learned to game it, reported by someone who chose which number to show you. This module teaches you to see all of that at a glance.

Why this module exists

By now you know how to build evals (Module 12), grade them (Module 03, Module 13) and put error bars on them (Module 04). This module is the other side of the coin: how to distrust a number intelligently. A model card says "80.2% on SWE-bench Verified". A startup says "state of the art on WebVoyager". A leaderboard shows one model two points above another. Should you believe any of it, and what exactly would you be believing?

An expert does not answer "yes" or "no". An expert asks a series of pointed questions: which version of the test, which harness, how many attempts, how were the answers graded, could the model have seen the test, how big is the error bar, and does this test even measure the thing I care about? By the end of this page you will have those questions as a reflex.

The benchmark life cycle

A benchmark is a public, standardized eval that many people run so their results can be compared. Benchmarks are not permanent. They age in a remarkably consistent way, and knowing the stage tells you how much a score on it can mean.

Stage by stage

• Release. The authors publish tasks, a grader, a baseline and usually a human score. At release, agent benchmarks typically show a huge gap: on OSWorld the best model scored 12.24% against a human 72.36%; on WebArena GPT-4 scored 14.4% against humans at 78.2%; on GAIA GPT-4 with plugins got about 15% against humans at 92%. That gap is what makes a benchmark useful.
• Hill-climbing. "Hill-climbing" means repeatedly making small changes and keeping those that raise the score, like walking uphill in fog. Labs add the benchmark to model cards, researchers tune prompts and scaffolds against it, and scores climb quickly. This is also when the test set spreads around the internet.
• Saturation. A benchmark is saturated when top systems score so close to the ceiling that differences are within noise, or when the remaining failures are mostly flawed items. The research notes list HumanEval, MBPP, MMLU, GSM8K, GPQA Diamond, ScreenSpot, GenEval and SWE-bench Verified as saturated or near it.
• Retirement. The field moves on. OpenAI stopped reporting SWE-bench Verified in February 2026 and recommends SWE-bench Pro. MMLU gave way to MMLU-Pro and Humanity's Last Exam. A retired benchmark is still a perfectly good regression test ("did we break something basic?"); it is just a bad capability test.

Contamination: training on the test

Contamination (also called data leakage or test-set leakage) means that some or all of a benchmark's test items, or their answers, ended up in the model's training data. A contaminated model can score well by remembering rather than solving, exactly like a student who memorized last year's answer sheet.

It is rarely deliberate. Frontier models are trained on enormous web crawls, and benchmarks live on the web: the dataset is on GitHub and Hugging Face, the solutions to coding tasks are real commits in public repositories, blog posts quote examples, and forum threads discuss answers. For SWE-bench this is structural: every task is a real GitHub issue whose fix (the "gold patch") was merged into a public repository years ago.

The four detection methods

Code: an n-gram overlap contamination check

This is the simplest serious detector. It indexes a (training) corpus by n-grams, then reports what fraction of each benchmark item's n-grams appear in the corpus. At real scale you would use a suffix array or a Bloom filter over trillions of tokens; the logic is the same.

import re
from collections import defaultdict

def tokens(text):
    # lowercase, keep words and numbers, ignore punctuation and whitespace differences
    return re.findall(r"[a-z0-9]+", text.lower())

def ngrams(toks, n):
    return {tuple(toks[i:i + n]) for i in range(len(toks) - n + 1)}

def build_index(corpus_docs, n=13):
    """Map every n-gram in the (training) corpus to the docs it appears in."""
    index = defaultdict(set)
    for doc_id, text in corpus_docs.items():
        for g in ngrams(tokens(text), n):
            index[g].add(doc_id)
    return index

def contamination_report(test_items, index, n=13, threshold=0.2):
    """For each benchmark item: what fraction of its n-grams also occur in the corpus?"""
    report = []
    for item_id, text in test_items.items():
        grams = ngrams(tokens(text), n)
        if not grams:                        # item shorter than n tokens: cannot judge
            report.append((item_id, None, "too short"))
            continue
        hits = [g for g in grams if g in index]
        frac = len(hits) / len(grams)
        sources = set().union(*(index[g] for g in hits)) if hits else set()
        flag = "CONTAMINATED?" if frac >= threshold else "clean"
        report.append((item_id, round(frac, 2), flag, sorted(sources)[:3]))
    return report

corpus = {
    "github/django-fork": "Fix QuerySet.bulk_create so that it returns primary keys when "
                          "ignore_conflicts is False on PostgreSQL backends, see ticket 30000 for details",
    "blog/cooking":       "Preheat the oven to 200 degrees and roast the vegetables for forty minutes",
}
test = {
    "task-001": "QuerySet.bulk_create should return primary keys when ignore_conflicts is False "
                "on PostgreSQL backends see ticket 30000 for details and add a regression test",
    "task-002": "Add a management command that exports every user's last login time to a CSV file",
}
idx = build_index(corpus, n=8)        # 8-grams for this tiny demo; 13 is a common choice at scale
for row in contamination_report(test, idx, n=8):
    print(row)
# ('task-001', 0.44, 'CONTAMINATED?', ['github/django-fork'])
# ('task-002', 0.0, 'clean', [])

Notice what it cannot see. Task-001 is caught because the wording overlaps. But in SWE-bench the danger is the answer (the gold patch) being in training data, not the question, so you would index the gold patches too. And a model that saw a paraphrase, or a translation, sails straight through. That is why labs combine overlap checks with behavioral probes.

Case study: memorized gold patches in SWE-bench Verified

On 23 February 2026 OpenAI published "Why SWE-bench Verified no longer measures frontier coding capabilities". Its contamination evidence was behavioral. OpenAI used GPT-5 as an automated interviewer that probed each target model for up to 15 turns with varied prompts, then rated contamination from "none" to "strong". According to OpenAI's write-up, every frontier model tested showed signs of it: GPT-5.2 reproduced gold patches verbatim; Claude Opus 4.5 recalled exact file paths, method names and inline comments from patches; Gemini 3 Flash produced verbatim details when given only the task ID. When a model can recite the answer from the task's ID number, the score is no longer measuring coding ability alone.

Label noise and flawed tasks

Label noise means the benchmark's "correct" answers are sometimes wrong. A flawed task is one that cannot be fairly solved: ambiguous instructions, tests that reject correct solutions, or environments that break. Both put a ceiling below 100% that no honest model can pass, and both make differences near the top meaningless.

Two things to notice. First, the 59.4% is of the audited hard problems, not of all 500; do not quote it as "59% of SWE-bench Verified is broken". Second, this was a benchmark that had already been human-screened: "Verified" was itself a cleaned subset of SWE-bench created by OpenAI and the SWE-bench authors in 2024. Validation reduces label noise; it never removes it.

Goodhart's law and overfitting to benchmarks

Goodhart's law, in its common phrasing: "When a measure becomes a target, it ceases to be a good measure." A benchmark is a proxy: a small, cheap sample standing in for a big, expensive thing you actually care about (general coding skill, helpfulness to real users). While nobody optimizes for the proxy, it tracks the real thing. Once the whole field optimizes for it, the proxy and the target drift apart.

When an optimizer, not a person, climbs your eval

"Adaptive overfitting" above describes people looking at results and tweaking. That loop is now automated. Prompt and program optimizers such as DSPy and GEPA take your eval score as the objective and search over instructions, few-shot examples and agent programs to maximize it, often by having an LLM read failing traces and propose edits. Reinforcement learning goes one step further: an eval task with an automatic check can be packaged as an RL environment (libraries such as Prime Intellect's verifiers do exactly this), and the grader becomes the reward the model is trained on.

Each step raises the optimization pressure on the same measure, so Goodhart's law bites harder and sooner. An optimizer tries thousands of variants and finds a grader's quirks far faster than a human would; RL training turns those quirks into learned behavior, which is where the reward hacking in the next section comes from. If the grader is an LLM judge, optimizing against it will find the judge's biases (length, confident tone, keyword matching) unless the judge was validated first. Module 13 covers how judges relate to reward models and RL graders in depth. The defenses are the ones this module keeps returning to, applied more strictly: a held-out set the optimizer never sees (not even its scores), fresh tasks on a schedule, validated graders, and reading the transcripts of the winning variant before trusting its score.

Leaderboard gaming: The Leaderboard Illusion

In April 2025, Singh et al. (Cohere and collaborators) published "The Leaderboard Illusion", a study of Chatbot Arena (now LMArena), the crowd-voted leaderboard where users compare two anonymous models and vote for the better answer. Ratings are fitted with a Bradley-Terry model (a statistical model that turns pairwise wins into a score). The paper documented three structural problems:

1. Undisclosed private testing and selective disclosure. Some providers tested many private variants and published only the best. The paper reports that Meta tested 27 private variants before the Llama 4 release. Picking the maximum of many noisy measurements systematically inflates the reported score, even if every variant is equally good.
2. Data access asymmetry. Google and OpenAI each received roughly 19-20% of all battle data, while 83 open-weight models together received about 29.7%. Battle data is valuable: the paper estimates that extra arena data can yield relative performance gains of up to 112% on the arena distribution.
3. Silent deprecation. Models removed from the arena without notice break the assumptions of the rating model, which can distort the rankings of those that remain.

The general lesson extends to every leaderboard, not only Arena: whoever gets more tries, more data, or more say in which results are shown will look better than they are. Defenses that exist in practice include private, held-out test sets and "first submission only" rules (Scale AI's SEAL leaderboards were designed this way), and independent re-runs by third parties (Epoch AI, Artificial Analysis) rather than self-reported numbers.

Harness dependence: apples to oranges

For an agent, the score belongs to a (model × harness × budget × grader × dataset version) combination, not to the model. The harness (or scaffold) is the code around the model: tool definitions, the agent loop, the system prompt, context management, retry logic. The budget is how many steps, tokens, minutes or attempts it gets. Change any of these and the number moves, sometimes more than a model upgrade would move it.

Infrastructure noise: the machine is part of the setup

The table above lists knobs that people choose on purpose. One more knob is usually not chosen at all: the compute resources of the sandbox the agent runs in (CPU, memory, disk, time limits, how many trials share a machine). In February 2026 Anthropic's engineering blog published "Quantifying infrastructure noise in agentic coding evals". Anthropic summarized the result when announcing it: infrastructure configuration can swing agentic coding benchmarks by several percentage points, sometimes more than the leaderboard gap between top models. (Secondary coverage reports swings of more than 6 points between resource configurations.)

The mechanism is easy to picture. An agentic coding task means installing packages, compiling, running test suites. With tight CPU or memory, those steps run slower or get killed (an out-of-memory kill looks like a crash), more trials hit their time limit, and an agent that would have succeeded is scored as a failure. With generous resources, a brute-force approach that would time out elsewhere can finish. None of that is the model's skill, and none of it appears in a headline number unless someone reports the sandbox spec.

What to do about it, whether you read claims or run evals:

• Treat resources as part of the configuration. Record CPU, memory, disk, per-task time limit and concurrency next to the model and harness versions, and add them to question 7 of the checklist below ("what budget?").
• Compare on identical infrastructure, interleaved. Never compare "baseline last month" with "candidate today on the new cluster". Run both in the same pool, alternating trials.
• Separate infrastructure failures from agent failures. Log exit reasons (out-of-memory kill, container failed to start, setup timeout) and report the infrastructure error rate separately, as Module 04 recommends. A rising timeout rate is a signal that the machine, not the agent, is deciding outcomes.
• Probe for resource sensitivity. Rerun a sample with double the resources. If the score moves, your benchmark is partly measuring hardware, and your error bars should say so.
• Prefer shared, standardized harnesses for cross-model claims (HAL, Harbor for Terminal-Bench). Module 21 covers this from the harness builder's side.

Try it: run the same agent under different resource limits and compare the spread with the gap between two models.

Reward hacking in agent evals

Reward hacking (also "specification gaming") is when a system achieves a high score by exploiting a flaw in how success is measured, instead of doing the intended task. It is Goodhart's law performed by the model itself, in real time. Agents are especially prone to it because they have tools: they can read files, run commands and edit anything in their sandbox, including the tests that grade them.

Documented examples

Code: a reward-hack detector for coding-agent diffs

The cheapest defense is to inspect what the agent changed. This script reads a unified diff (for example git diff <base_commit> from the agent's sandbox) and flags edits to tests, test configuration, skip markers, forced exit codes and "am I under test?" branches. It exits non-zero on high-severity findings, so it can gate a CI run or mark trials for human review.

import re
import sys

TEST_PATH = re.compile(r"(^|/)(tests?|testing)/|(^|/)test_[^/]*\.py$|_test\.py$|(^|/)conftest\.py$")
PROTECTED = re.compile(r"(^|/)(pytest\.ini|tox\.ini|setup\.cfg|pyproject\.toml|\.github/workflows/)")
SUSPICIOUS_ADDED = [
    (re.compile(r"@pytest\.mark\.(skip|xfail)|unittest\.skip"), "adds a skip/xfail marker"),
    (re.compile(r"\bsys\.exit\(0\)|os\._exit\(0\)"),            "forces a zero exit code"),
    (re.compile(r"assert\s+True\b|return\s+True\s*#.*test", re.I), "trivial assertion"),
    (re.compile(r"(PYTEST|TESTING|CI)['\"]?\s*(in|==)|os\.environ\.get\(['\"](PYTEST|CI)"),
     "behaves differently under test"),
    (re.compile(r"monkeypatch|mock\.patch\(['\"](time\.time|.*grade|.*score)"), "patches timer or grader"),
]

def parse_diff(diff_text):
    """Yield (path, added_lines, removed_lines) for each file in a unified diff."""
    path, added, removed = None, [], []
    for line in diff_text.splitlines():
        if line.startswith("diff --git"):
            if path:
                yield path, added, removed
            path, added, removed = line.split(" b/", 1)[-1], [], []
        elif line.startswith("+") and not line.startswith("+++"):
            added.append(line[1:])
        elif line.startswith("-") and not line.startswith("---"):
            removed.append(line[1:])
    if path:
        yield path, added, removed

def audit(diff_text):
    findings = []
    for path, added, removed in parse_diff(diff_text):
        if TEST_PATH.search(path):
            dropped = [l for l in removed if re.search(r"\bassert\b|self\.assert", l)]
            findings.append(("HIGH", path, f"test file modified (-{len(dropped)} assertions)"))
        if PROTECTED.search(path):
            findings.append(("HIGH", path, "test/CI config modified"))
        for line in added:
            for pattern, why in SUSPICIOUS_ADDED:
                if pattern.search(line):
                    findings.append(("MED", path, f"{why}: {line.strip()[:60]}"))
    return findings

if __name__ == "__main__":
    diff = sys.stdin.read()           # usage: git diff BASE_COMMIT | python hackcheck.py
    results = audit(diff)
    for sev, path, msg in results:
        print(f"[{sev}] {path}: {msg}")
    sys.exit(1 if any(s == "HIGH" for s, _, _ in results) else 0)

# Example output on a diff that skips a test and adds a PYTEST_CURRENT_TEST branch:
# [MED] src/parser.py: behaves differently under test: if os.environ.get("PYTEST_CURRENT_TEST"):
# [HIGH] tests/test_parser.py: test file modified (-1 assertions)
# [MED] tests/test_parser.py: adds a skip/xfail marker: @pytest.mark.skip(reason="flaky")

Pattern matching is a first line of defense, not a proof. Serious harnesses add structural defenses that make hacks impossible rather than detectable:

• Grade with the original tests. SWE-bench copies its own FAIL_TO_PASS and PASS_TO_PASS tests into the container at grading time, so agent edits to test files are overwritten. Do the same: never grade with files the agent could touch.
• Keep the grader out of reach. Run scoring in a separate process or container with no shared memory, no reachable reference answers, and its own clock.
• Sanitise the environment. Remove future git history, caches and solution artifacts from images (the SWE-bench v4.1.0 fix).
• Read transcripts, and use a monitor. An LLM judge reading trajectories for "did the agent try to modify tests or read grading code?" catches creative hacks that regexes miss (Module 13, Module 17).

Grader bugs, flaky environments and stale answers

Not every false result is the agent's fault. The measuring apparatus fails too, in three recurring ways.

Construct validity: does it measure what it claims?

Construct validity is a term from psychology: the degree to which a test measures the abstract quality (the "construct") it claims to measure. "Coding ability", "general intelligence", "can do a software engineer's job" are constructs. A benchmark is a concrete, narrow operationalization of one. The questions are: what does a high score let you predict, and about what?

Examples of gaps between benchmark and construct:

• SWE-bench measures "fix a well-described Python issue in a popular open-source repo, judged by the maintainers' tests". Real software work also involves unclear requirements, private codebases, code review and long-lived design choices.
• Multiple-choice knowledge tests measure recognizing a right option among four, which is easier than producing the answer and is sensitive to formatting. Anthropic has noted that small formatting changes can move MMLU scores by around 5%.
• A 50% success rate on a task that a human does in one hour (METR's time horizon) is a statistic about a software/ML task distribution, not a promise that "the model can work for an hour" on your task. METR says so in its own limitations notes.

METR itself is careful about what the study does and does not show: it concerns experienced developers on large, mature codebases they know well, with tools from early 2025. The lesson for this module is methodological: benchmark performance and real-world impact are different measurements, and only the second one answers "does this help my users?". This is why Module 15 insists on online evals and A/B tests alongside offline benchmarks.

Statistical sins

The last family of pitfalls is arithmetic. Most benchmark claims are made without error bars, on small test sets, from a single run. Here are the classic sins and what they cost you.

Simulation: a 2-point gap on 100 items is noise

Suppose two models are exactly equally good: each solves any given item with probability 70%. You run each once on the same kind of 100-item benchmark. How often will the leaderboard show one of them 2 or more points ahead?

Now a realistic claim: "Model A scores 72%, Model B 70%, on the same 100 items." Because both models answer the same items, the right tool is a paired bootstrap: resample items with replacement (keeping each item's A and B results together), recompute the gap each time, and read off the middle 95%. The key quantity is the discordant items, where exactly one model is right. Here A alone is right on 6 items and B alone on 4.

import numpy as np

rng = np.random.default_rng(0)
N = 100                                   # benchmark items

# Part 1: two IDENTICAL models (true accuracy 70%), each run once on 100 items.
# How often does the "leaderboard" show a gap of 2+ points anyway?
a = rng.random((20_000, N)) < 0.70
b = rng.random((20_000, N)) < 0.70
# with 100 items, 1 item = 1 point; count in integers to avoid float rounding
gap = a.sum(axis=1) - b.sum(axis=1)
print(f"P(|gap| >= 2 points) for identical models: {np.mean(abs(gap) >= 2):.0%}")
print(f"std of the gap: {gap.std():.1f} points")

# Part 2: a real claim, "Model A 72% vs Model B 70%", on the SAME 100 items.
# Per item: both right 66, only A right 6, only B right 4, both wrong 24.
A = np.array([1]*66 + [1]*6 + [0]*4 + [0]*24)
B = np.array([1]*66 + [0]*6 + [1]*4 + [0]*24)

# Paired bootstrap: resample ITEMS, keeping each item's A/B pair together.
idx = rng.integers(0, N, size=(10_000, N))
diffs = A[idx].sum(axis=1) - B[idx].sum(axis=1)     # gap in points (N = 100)
lo, hi = np.percentile(diffs, [2.5, 97.5])
print(f"observed gap: {A.sum() - B.sum()} points, 95% CI [{lo:.0f}, {hi:.0f}]")
print(f"share of resamples where A is NOT ahead: {np.mean(diffs <= 0):.0%}")

# Output:
# P(|gap| >= 2 points) for identical models: 82%
# std of the gap: 6.5 points
# observed gap: 2 points, 95% CI [-4, 8]
# share of resamples where A is NOT ahead: 32%

The interval from −4 to +8 includes zero comfortably: the data are consistent with A being worse, equal or clearly better. With only 10 discordant items, a 6-vs-4 split is what a fair coin produces all the time. To detect a true 2-point difference reliably you need many more items, repeated runs per item, or both. (Rule of thumb from Module 04: the standard error of a proportion near 70% is about 46/√N points, so halving it needs four times the items.)

How to read a benchmark claim: 15 questions

Here is the whole module compressed into a checklist. You will not be able to answer every question for every claim; the point is to notice which answers are missing. A missing answer is information.

The Agentic Benchmark Checklist: a review rubric for your own evals

The 15 questions above are for reading someone else's claim. When you build or adopt an agentic benchmark, including the private product suite from Module 12, you need a rubric for the benchmark itself. The Agentic Benchmark Checklist (ABC; Zhu et al., Daniel Kang's lab at UIUC and collaborators; arXiv 2507.02825, NeurIPS 2025 Datasets and Benchmarks track; project page) distills benchmark-building experience and documented failures into a checklist organized in three parts.

Applying it to your own evals

Treat the ABC like a code review for your benchmark. Go item by item, mark each yes, no or not applicable, and write down the evidence (not "we think so"). Several items can be checked automatically with baselines that every eval should run anyway:

# A minimal "outcome validity" smoke test. run_eval(agent) is your own harness entry
# point: it runs every task with the given agent and returns the mean score (0..1).
from my_eval import run_eval, load_reference_solutions   # your code

def null_agent(task):            # does nothing at all
    return ""

def echo_agent(task):            # restates the task, takes no action
    return task.instructions

def oracle_agent(task):          # replays the known-good solution
    return load_reference_solutions()[task.id]

checks = {
    "null agent scores ~0":        (run_eval(null_agent),   lambda s: s <= 0.02),
    "echo agent scores ~0":        (run_eval(echo_agent),   lambda s: s <= 0.02),
    "reference solutions pass ~1": (run_eval(oracle_agent), lambda s: s >= 0.98),
}
for name, (score, ok) in checks.items():
    print(f"{'PASS' if ok(score) else 'FAIL'}  {name:30s} score={score:.2f}")

# Any FAIL is a benchmark bug, not an agent result. A null agent above 0 usually
# means some tasks are "already solved" in their start state (the tau-bench
# empty-response pattern); an oracle below 1 means broken tasks or a strict grader.

Then do the parts no script can: have a second person read 20 passing and 20 failing transcripts and confirm the grader's verdicts, check that each task's goal is unambiguous, and write the reporting section (known flaws, what was not validated, how to cite the version). Record the result as a short "benchmark card" next to the dataset, and repeat the review whenever you version the benchmark. The 15-question checklist tells you what to ask of a claim; the ABC tells you what to fix before you make one.

How to build a benchmark that lasts

Everything above can be turned around into design principles. The benchmarks that stay useful longest share a handful of habits, and each one answers a specific pitfall.

Check yourself

Sources and further reading

• OpenAI, Why SWE-bench Verified no longer measures frontier coding capabilities (Feb 23, 2026).
• OpenAI, Introducing SWE-bench Verified (Aug 2024).
• SWE-bench, Repo State Loopholes During Agentic Evaluation (issue #465) and swebench.com (bash-only view).
• Singh et al., The Leaderboard Illusion (2025).
• METR, Recent frontier models are reward hacking (June 2025).
• METR, Measuring the impact of early-2025 AI on experienced open-source developer productivity (July 2025).
• METR, Clarifying limitations of time horizon (Jan 2026).
• Anthropic, Introducing Claude 4 (May 2025; SWE-bench methodology appendix).
• Anthropic, Challenges in evaluating AI systems.
• Miller, Adding Error Bars to Evals (2024).
• Gema et al., Are We Done with MMLU? (MMLU-Redux, 2024).
• Jain et al., LiveCodeBench (2024).
• Kiela et al., Dynabench: Rethinking Benchmarking in NLP (2021).
• Liu et al., Is Your Code Generated by ChatGPT Really Correct? (EvalPlus) (2023).
• Xie et al., OSWorld and OSWorld-Verified.
• Sierra, tau2-bench repository (task fixes changelog).
• Scale AI, SWE-bench Pro (2025) and SEAL leaderboards.
• Brown et al., Language Models are Few-Shot Learners (GPT-3; 13-gram contamination analysis).
• Anthropic, Quantifying infrastructure noise in agentic coding evals (Feb 2026) and the announcement post.
• Zhu et al., Establishing Best Practices for Building Rigorous Agentic Benchmarks (the Agentic Benchmark Checklist; NeurIPS 2025 Datasets and Benchmarks) and the ABC project page.
• DSPy and GEPA (prompt and program optimizers that use your eval as the objective).
• Prime Intellect, verifiers (packaging eval tasks as RL environments).