Evaluating coding agents

Code is the friendliest domain in all of agent evaluation, because you can run the answer. It is also where the field learned, the hard way, that "the tests passed" and "the agent solved the problem" are not the same sentence.

Why code is the easy case (and the trap)

In most of this course, the hardest part of an eval is the grader: how do you decide, automatically, whether an answer is good? For a customer-support reply you may need an LLM judge or a human (Module 03). For code, you have something much better: you can execute the program and check what it does. This is called execution-based grading, and it is the reason coding became the flagship domain for agent evaluation.

Execution-based grading has three wonderful properties:

• It accepts many correct answers. Two completely different implementations both pass if they behave the same. You do not need the output to match a reference string.
• It is cheap and deterministic (mostly). Running a test suite costs seconds, not a human's afternoon.
• It is hard to argue with. A failing test is concrete evidence, not a judge's opinion.

And one big trap: the tests are only as good as whoever wrote them. If the tests are thin, wrong code passes. If they are too strict, correct code fails. If the agent can see or guess the tests, it can pass without solving anything. Most of the expert material in this module is about that trap.

Three generations of coding evals

Coding benchmarks have grown in three big steps. Each step made the task more realistic, more expensive to run, and harder to grade well.

Generation 1: function-level benchmarks

HumanEval

HumanEval (Chen et al., OpenAI, 2021, the "Codex paper") is 164 hand-written Python problems. Each gives the model a function signature and a docstring (the comment that says what the function should do). The model writes the body. A hidden check() function with, on average, about 7.7 unit tests decides whether it is right.

The metric the Codex paper introduced is pass@k: the probability that at least one of k sampled answers is correct. Naively you would sample k answers and check, but that is very noisy. The paper's trick is to sample n ≥ k answers, count c correct, and compute an unbiased estimate:

pass@k = 1 − C(n−c, k) / C(n, k), averaged over problems.

(C(a, b) is "a choose b", the number of ways to pick b items from a.) In words: of all the ways to pick k samples from your n, what fraction contain no correct one? Subtract that from 1.

import numpy as np

def pass_at_k(n: int, c: int, k: int) -> float:
    """Unbiased pass@k from the Codex paper (Chen et al., 2021).
    n = samples drawn, c = how many passed the tests, k = budget."""
    if n - c < k:          # every k-subset must contain a correct sample
        return 1.0
    # numerically stable form of 1 - C(n-c, k) / C(n, k)
    return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))

print(pass_at_k(10, 3, 1))   # 0.30
print(pass_at_k(10, 3, 5))   # 0.9167

MBPP, and why both are now "solved"

MBPP (Austin et al., Google, 2021) is 974 crowd-sourced, entry-level Python problems, each with 3 assert tests, plus a cleaned "sanitized" subset of 427. Same idea, easier problems.

Both are now considered saturated (frontier models score near the ceiling) and contaminated (the problems and solutions have been on the public internet for years, so they are almost certainly in training data). They also have too few tests: the EvalPlus project (Liu et al., 2023) added about 80 times more tests to make HumanEval+ and MBPP+, and scores dropped, meaning some "passing" solutions had been wrong all along. That last finding is worth remembering: adding tests lowered the scores. Weak tests silently inflate every number built on them.

Function-level benchmarks are still useful for quick sanity checks and for small models, but they tell you almost nothing about whether an agent can work in a real codebase. For that, we need generation 2.

Generation 2: SWE-bench, exactly how it works

SWE-bench (Jimenez, Yang et al., Princeton, ICLR 2024) changed the question from "can you write a function?" to "can you fix a real bug in a real project?". It contains 2,294 task instances built from real GitHub issues and the pull requests that fixed them, drawn from 12 popular Python repositories (Django, scikit-learn, sympy, and others).

A few terms first:

• Issue: a bug report or feature request on GitHub, written by a human, in prose.
• Pull request (PR): the set of code changes that fixed it, usually with new or changed tests.
• Commit: a snapshot of the repository at one moment. The base commit is the snapshot just before the fix.
• Patch / diff: a text description of changes to files ("remove line 12, add these 3 lines"). git apply applies a patch to a checkout.
• Docker container: a lightweight isolated Linux environment with exactly the right Python version and dependencies installed, built from a recipe called an image.

Each PR is split into two parts. The gold patch is the change to the source code: the human's fix, kept as a reference answer. The test patch is the change to the test files: kept secret and used for grading. The agent sees only the issue text and the repository at the base commit.

FAIL_TO_PASS and PASS_TO_PASS

These two lists are the heart of SWE-bench grading. Both are computed once, when the benchmark is built, by running the tests with and without the gold patch:

• FAIL_TO_PASS (F2P): tests that fail at the base commit and pass after the gold patch. These encode "the bug is fixed" or "the feature now works". Usually they are the new tests in the test patch.
• PASS_TO_PASS (P2P): tests that pass both before and after the gold patch. These encode "you didn't break anything else". They are the regression check.

The agent's patch counts as resolved only if every F2P test passes and every P2P test still passes. It is all-or-nothing per instance: fixing half the bug earns nothing.

What an instance and a prediction look like

The dataset lives on Hugging Face. Each row is one task instance. The field names below are the real ones, and they are the same across SWE-bench, Lite and Verified.

from datasets import load_dataset
import json

ds = load_dataset("princeton-nlp/SWE-bench_Verified", split="test")   # 500 rows
row = ds[0]

row["instance_id"]        # e.g. "astropy__astropy-12907"
row["repo"]               # "astropy/astropy"
row["base_commit"]        # the commit to check out BEFORE the fix
row["problem_statement"]  # the GitHub issue text: this is the task prompt
row["patch"]              # gold source patch (reference answer, never shown)
row["test_patch"]         # hidden test changes, applied only at grading time
json.loads(row["FAIL_TO_PASS"])   # list of test ids, stored as a JSON string
json.loads(row["PASS_TO_PASS"])   # list of test ids that must keep passing

# Your agent produces one JSON object per instance, written to preds.jsonl:
pred = {
    "instance_id": row["instance_id"],
    "model_name_or_path": "my-agent-v3",
    "model_patch": "diff --git a/astropy/modeling/separable.py ...",
}

Then the official harness builds the Docker images, applies each patch, runs the tests and writes a report:

pip install swebench
python -m swebench.harness.run_evaluation \
    --dataset_name princeton-nlp/SWE-bench_Verified \
    --predictions_path preds.jsonl \
    --max_workers 8 \
    --run_id my-agent-v3
# -> per-instance logs + a JSON report listing resolved / unresolved / error ids

The SWE-bench family

SWE-bench spawned a family of variants. Each fixes a specific problem with the one before.

Why Verified was created

In 2024, OpenAI's Preparedness team noticed that many SWE-bench failures were not the agent's fault. They had 93 professional Python developers annotate 1,699 randomly sampled SWE-bench instances, with three independent reviews each. The findings:

Two kinds of problem dominated:

Annotators also estimated difficulty in time buckets; fewer than 10% of Verified tasks were estimated to take an experienced engineer more than an hour. That is worth remembering: Verified is mostly short tasks.

How a coding score can lie

A "percent resolved" number can be wrong in two directions. It can be inflated (the agent gets credit it did not earn) or deflated (the agent is marked wrong when it was right). An expert reads every coding leaderboard with both columns in mind.

Inflation: getting credit you didn't earn

• Contamination. SWE-bench tasks come from public GitHub repos, and the fixes have been public for years. In February 2026 OpenAI reported that all frontier models it tested could reproduce gold patches or verbatim problem-statement details for some Verified tasks (for example, quoting inline comments from the original diff). A model that remembers the answer is not demonstrating problem solving.
• Solution leakage in the issue. Some issue reporters paste a suggested fix, or even a full diff, into the issue. The agent then only has to copy it. Researchers have flagged Verified instances where the solution is reachable directly from the problem statement (SWE-bench GitHub issue #669).
• Git history leakage. The Docker images originally contained the full git repository, including commits and tags created after the base commit. Agents discovered they could run commands like git log --all and read the future fix. SWE-bench issue #465 (2025) documents trajectories from several model families doing exactly this; the maintainers rebuilt images to strip future branches, tags and reflogs in release 4.1.0. A similar report was filed against SWE-bench Pro's open-source set. The lesson is general: anything reachable from inside the sandbox is part of the prompt.
• Weak tests. If the F2P tests cover only the exact example from the issue, a patch that special-cases that example passes. The UTBoost work argued that Verified still has instances whose tests are insufficient to reject wrong patches.

Deflation: failing correct work

• Over-strict and off-spec tests. OpenAI's 2026 audit looked at 138 Verified problems that o3 did not consistently solve over 64 runs. Engineers found material flaws in 59.4% of them: 35.5% had tests enforcing specific implementation details, 18.8% tested functionality the issue never described, and 5.1% had other problems. OpenAI stopped reporting Verified and recommended SWE-bench Pro instead.
• Environment problems. A dependency that no longer installs, a test that needs network access, a timeout on a slow machine. These look like agent failures in the aggregate number.
• Harness limits. The same model with a 30-step budget and with a 300-step budget are different systems. More on this below.

Generation 3: terminals, research, and money

SWE-bench measures "fix this issue in this repo". Real engineering also means setting up environments, running long jobs, debugging builds, training models and working for hours. Generation 3 benchmarks give the agent a whole machine and a longer goal.

Terminal-Bench

Terminal-Bench (Stanford and the Laude Institute) gives an agent a Linux terminal inside a Docker container and a task in plain English: compile this project, fix this server config, train this small model, recover this corrupted file. The 2.0 version described in the January 2026 paper has 89 tasks across software engineering, system administration, ML, security, bioinformatics and more, each with a human-written reference solution and test scripts. Scoring is binary pass/fail from the tests, usually averaged over several runs, and results are reported for model × harness pairs (for example the Terminus reference agent, Claude Code, or Codex CLI). At paper time, "frontier models and agents score less than 65%".

Tasks are run with the Harbor framework. Every task is a folder with a fixed layout:

A test script is usually just pytest checking the end state of the container (a file exists with the right contents, a server responds, a model's accuracy is above a threshold), then writing the reward:

#!/bin/bash
# tests/test.sh  (runs inside the task container after the agent stops)
cd /app
if pytest /tests/test_outputs.py -q; then
  echo 1 > /logs/verifier/reward.txt
else
  echo 0 > /logs/verifier/reward.txt
fi

# Run Terminal-Bench 2.0 with a given agent harness and model (Harbor README)
harbor run --dataset terminal-bench@2.0 --agent claude-code \
           --model anthropic/claude-opus-5-5 --n-concurrent 4

Because the grader checks the state of the machine, not a diff, Terminal-Bench is a natural bridge to the computer-use evals of Module 08. Its known weaknesses: results are sensitive to the harness, some tasks can be flaky (network, timeouts), and 89 tasks gives wide confidence intervals (a single task is more than one percentage point).

Aider polyglot

The Aider polyglot benchmark (Paul Gauthier, Dec 2024) uses 225 of the hardest Exercism exercises in C++, Go, Java, JavaScript, Python and Rust. The model edits code through Aider's edit formats and gets two attempts: after a first failure, it sees the unit-test output and tries again. It reports percent solved, percent well-formed edits (did the model follow the edit format at all?) and cost. It is a good example of measuring the interface skill (producing edits a tool can apply) as well as the coding skill. Exercism solutions are public, so contamination is a real risk.

LiveCodeBench: fighting contamination with time

LiveCodeBench (Jain et al., 2024) continuously collects new competitive-programming problems from LeetCode, AtCoder and Codeforces and tags each with its release date. You evaluate a model only on problems published after its training cutoff, so it cannot have seen them. This is a contamination-aware time window. It covers code generation, self-repair, code execution and test-output prediction, scored with pass@1. The idea generalizes: SWE-rebench and SWE-bench-Live apply the same freshness logic to repository tasks.

Research-engineering benchmarks: MLE-bench, RE-Bench, PaperBench

• MLE-bench (OpenAI, Oct 2024): 75 Kaggle competitions run offline. The agent must train models and submit predictions, which are graded against each competition's private leaderboard. The metric is the percentage of competitions where the agent would have earned a medal. At release, o1-preview with the AIDE scaffold earned a medal in 16.9% of competitions. Runs are expensive (up to 24 hours on a GPU per competition).
• RE-Bench (METR, Nov 2024): 7 open-ended ML research-engineering environments, such as optimizing a GPU kernel or fine-tuning a small model. Scores are continuous, normalized so the starting solution is 0 and a reference solution is 1, and compared against 71 eight-hour attempts by 61 human experts. Agents did better than humans at 2-hour budgets; humans won at 8 to 32 hours. It feeds METR's time-horizon measurements (Module 11).
• PaperBench (OpenAI, Apr 2025): replicate 20 ICML 2024 Spotlight and Oral papers from scratch. There are 8,316 gradable sub-tasks in hierarchical rubrics co-written with the papers' authors, graded by an LLM judge that was itself validated on a judge-eval set. At release, the best agent (Claude 3.5 Sonnet, new) averaged 21.0%; ML PhDs reached 41.4% on a subset. This is what execution-based grading looks like when the "answer" is too big to test with asserts: you fall back to a rubric and a judge (Module 13).

SWE-Lancer: pricing the work

SWE-Lancer (OpenAI, Feb 2025) uses more than 1,400 real freelance tasks from Upwork on the Expensify codebase, worth $1M in actual payouts. Two task types:

• IC SWE (individual contributor) tasks: implement the fix. Graded by end-to-end Playwright tests (browser automation that clicks through the real app) that were triple-verified by engineers. End-to-end tests are much harder to game than unit tests because they check behavior a user would see.
• SWE Manager tasks: choose the best of several implementation proposals. Graded against the choice the original engineering manager made.

Its headline metric is dollars earned: each task is weighted by what it actually paid. The public "Diamond" split is worth $500.8K; at release, Claude 3.5 Sonnet earned $208K of it. Dollar weighting is a neat idea: it automatically gives hard, valuable tasks more weight than trivial ones, which an unweighted percentage does not.

Coding benchmarks side by side

Contamination ratings are this course's judgment based on each benchmark's construction, not a measured value. Sources for sizes and graders are listed at the end of the module. Deliberately, there are no leaderboard numbers here: they change monthly, and several are no longer meaningful (see the Verified audit above).

Execution-based grading done right

Whether you use a public benchmark or build your own, the same engineering rules make execution-based grading trustworthy.

Hidden tests

The tests used for grading must not be visible to the agent during its run. Otherwise the agent can (and, under optimization pressure, will) write code that satisfies the tests rather than the task, or simply edit the tests. SWE-bench applies test_patch only in a fresh grading container; Harbor copies tests/ in only at verification time. Your agent may of course write and run its own tests; that is good engineering. It just must not see the grader's.

Flaky tests

A flaky test passes or fails on the same code depending on timing, randomness, network or ordering. Flaky tests add noise to both F2P and P2P, and noise can look like a regression or an improvement. Standard defenses:

• When building the benchmark, run the gold solution several times (3 to 5) and drop any test whose result varies.
• Run the grader with network disabled and random seeds fixed, so behavior cannot depend on the outside world.
• Track per-test flip rates across runs; quarantine tests that flip.
• Never "re-run failed tests until they pass" only for the agent: that biases the score upward. Whatever policy you choose, apply it identically to every system.

Sandboxing

Coding agents execute arbitrary code. That code may be buggy, may delete files, or may (under adversarial conditions) try to escape. The sandbox is both a safety boundary and a measurement boundary: it defines exactly what the agent can see and do.

Here is a minimal grader that applies these rules with plain Docker. It assumes you already have an image per repository with dependencies installed.

import subprocess, tempfile, pathlib, re

def run(cmd, **kw):
    return subprocess.run(cmd, capture_output=True, text=True, **kw)

def run_tests(image: str, repo: str, commit: str, patches: list[str],
              test_files: list[str], timeout: int = 900) -> dict[str, bool]:
    """Fresh checkout at `commit`, apply patches in order, run pytest in a
    locked-down container. Returns {test_id: passed}."""
    with tempfile.TemporaryDirectory() as tmp:
        wt = pathlib.Path(tmp) / "repo"
        run(["git", "-C", repo, "worktree", "add", "--detach", str(wt), commit], check=True)
        try:
            for p in patches:
                r = run(["git", "-C", str(wt), "apply", "-"], input=p)
                if r.returncode != 0:
                    return {}                      # patch does not apply: unresolved
            r = run(["docker", "run", "--rm",
                     "--network", "none",          # no downloading the upstream fix
                     "--memory", "4g", "--cpus", "2",
                     "--user", "1000:1000",
                     "-v", f"{wt}:/repo", "-w", "/repo", image,
                     "python", "-m", "pytest", "-rA", "-p", "no:cacheprovider",
                     *test_files], timeout=timeout)
        finally:
            run(["git", "-C", repo, "worktree", "remove", "--force", str(wt)])
    # "-rA" prints one summary line per test, e.g. "PASSED tests/test_x.py::test_a"
    results = {}
    for status, test_id in re.findall(r"^(PASSED|FAILED|ERROR) (\S+)", r.stdout, re.M):
        results[test_id] = status == "PASSED"
    return results

def grade(inst: dict, model_patch: str, image: str, repo: str) -> bool:
    """SWE-bench-style resolve rule."""
    res = run_tests(image, repo, inst["base_commit"],
                    [model_patch, inst["test_patch"]],   # hidden tests applied LAST
                    inst["test_files"])
    f2p_ok = all(res.get(t, False) for t in inst["FAIL_TO_PASS"])
    p2p_ok = all(res.get(t, False) for t in inst["PASS_TO_PASS"])
    return f2p_ok and p2p_ok

Beyond pass/fail

"Tests pass" is necessary but not sufficient. Two patches can both resolve an issue while one is a clean three-line fix and the other rewrites a module, deletes a warning, adds a dependency and costs ten times as much. For your own evals, record a scorecard per task, not a single bit.

An LLM judge is the practical way to score the soft dimensions. Give it the issue, the diff, and a specific rubric with anchored levels (Module 13 covers why anchors matter). Keep correctness out of the judge's hands: the tests already decided that.

import json

RUBRIC = """You are reviewing a code change that ALREADY passes its tests.
Do not judge correctness. Score each criterion 1-3 using these anchors.

minimality:  3 = only lines needed for the issue; 2 = small unrelated edits;
             1 = rewrites or reformats code the issue did not concern
readability: 3 = clear names, follows the file's existing style;
             2 = understandable but awkward; 1 = hard to follow
robustness:  3 = handles the edge cases implied by the issue;
             2 = handles the reported case only; 1 = special-cases the example
             input (e.g. `if x == "foo": return ...`)
Return JSON: {"minimality": n, "readability": n, "robustness": n, "notes": "..."}"""

def judge_patch(issue: str, diff: str, call_llm) -> dict:
    """call_llm(system, user) -> str : any chat model client you use."""
    user = f"<issue>\n{issue}\n</issue>\n<diff>\n{diff}\n</diff>"
    scores = json.loads(call_llm(RUBRIC, user))
    return scores

def scorecard(passed: bool, diff: str, trace: dict, judge: dict) -> dict:
    added = sum(1 for l in diff.splitlines() if l.startswith("+") and not l.startswith("+++"))
    removed = sum(1 for l in diff.splitlines() if l.startswith("-") and not l.startswith("---"))
    return {
        "resolved": passed,
        "lines_changed": added + removed,
        "touched_tests": "tests/" in diff,          # crude scope check
        "cost_usd": trace["cost_usd"],
        "turns": trace["num_turns"],
        "wall_s": trace["wall_seconds"],
        **{f"judge_{k}": v for k, v in judge.items() if k != "notes"},
    }

Reporting cost next to accuracy

A worked example. Agent A resolves 120 of 200 tasks (60%) at an average of $0.40 per task. Agent B resolves 128 (64%) at $2.10 per task. Cost per resolved task: A is 200 × 0.40 / 120 = $0.67; B is 200 × 2.10 / 128 = $3.28. B is 4 points better (check whether that is significant with a paired test, Module 04: on 200 tasks it may not be) and about 5 times more expensive per success. Whether that is worth it is a product decision, but you can only make it if you measured both.

The scaffold matters as much as the model

A coding agent is model + harness. The harness (also called scaffold) is everything around the model: the system prompt, the tools (shell, file viewer, editor, search), how edits are applied, how much output is shown, how many steps are allowed, whether it retries, whether it runs tests before submitting. Change the harness and the score changes, sometimes more than a model upgrade would.

Documented examples:

• The SWE-agent paper (Yang et al., May 2024) showed that giving a model a purpose-built agent-computer interface (ACI) raised SWE-bench pass@1 to 12.5%, "far" beyond the previous state of the art from non-interactive, retrieve-then-patch approaches. Same class of model, very different result, because the agent could now navigate, edit safely and run code.
• Anthropic's January 2025 engineering post reported 49% on SWE-bench Verified for the upgraded Claude 3.5 Sonnet using a deliberately simple scaffold with just two tools (a Bash tool and an edit tool), beating the previous best of 45%. The post states that performance "can vary significantly based on this scaffolding, even when using the same underlying AI model".
• mini-SWE-agent reached 65% on Verified in July 2025 with about 100 lines of Python. SWE-bench's official "bash-only" leaderboard view runs every model in this same minimal harness, precisely so that model comparisons are not confounded by harness engineering.
• Terminal-Bench reports results per model × harness pair for the same reason.

For your own work, this cuts both ways. If you are choosing a model, hold the harness fixed and swap models. If you are building an agent product, hold the model fixed and treat the harness as the thing you are optimizing, with your eval as the scoreboard.

Build your own coding-agent eval from your repo's history

Public benchmarks tell you about general skill on popular open-source Python projects. What you want to know is whether an agent can work in your codebase: your frameworks, your conventions, your weird build. The good news: your git history is a ready-made benchmark generator. Every merged pull request that added tests is a task with a known answer and a built-in grader. This is exactly how SWE-bench was built, applied to your repo.

Step by step, here is what each stage does and why:

1. Collect merged PRs. Use the GitHub CLI or API. Linked issues make the best task descriptions, because they were written before the fix, by someone who did not know the answer.
2. Filter. Keep PRs that change both source and test files, that have a description a human could act on, and whose diff is a reasonable size. Drop dependency bumps, reformatting and pure refactors.
3. Split the diff by path into the gold source patch and the hidden test patch.
4. Label the tests by running them at the parent commit with only the test patch (new tests should fail), then with both patches (they should pass). Tests that fail then pass are F2P; tests that pass both times are P2P.
5. Validate. Throw away tasks with no F2P tests, tasks where the gold patch doesn't pass reliably, and tests that flip between runs.

"""Mine SWE-bench-style tasks from your own repository. A sketch: adapt the
path rules, test runner and Docker image to your project."""
import json, re, subprocess
from pathlib import Path

REPO = str(Path("~/src/myrepo").expanduser())
GH_REPO = "my-org/myrepo"
IMAGE = "myrepo-eval:latest"      # Dockerfile with your deps preinstalled

def sh(*cmd, **kw) -> str:
    return subprocess.run(cmd, capture_output=True, text=True, check=True, **kw).stdout

def is_test(path: str) -> bool:
    return path.startswith("tests/") or re.search(r"(^|/)test_[^/]+\.py$", path) is not None

def merged_prs(limit: int = 1000) -> list[dict]:
    out = sh("gh", "pr", "list", "--repo", GH_REPO, "--state", "merged",
             "--limit", str(limit),
             "--json", "number,title,body,mergeCommit,closingIssuesReferences")
    return json.loads(out)

def build_instance(pr: dict) -> dict | None:
    merge = pr["mergeCommit"]["oid"]
    base = sh("git", "-C", REPO, "rev-parse", f"{merge}^1").strip()  # pre-PR commit
    files = sh("git", "-C", REPO, "diff", "--name-only", base, merge).split()
    src = [f for f in files if f.endswith(".py") and not is_test(f)]
    tests = [f for f in files if is_test(f)]
    if not src or not tests or len(files) > 15:
        return None                                   # step 2: filter
    gold = sh("git", "-C", REPO, "diff", base, merge, "--", *src)
    test_patch = sh("git", "-C", REPO, "diff", base, merge, "--", *tests)

    # step 4: label tests (run_tests is the Docker helper from the grading section)
    before = run_tests(IMAGE, REPO, base, [test_patch], tests)
    after = run_tests(IMAGE, REPO, base, [gold, test_patch], tests)
    f2p = [t for t, ok in after.items() if ok and not before.get(t, False)]
    p2p = [t for t, ok in after.items() if ok and before.get(t, False)]

    # step 5: validate stability of the gold solution
    for _ in range(2):
        again = run_tests(IMAGE, REPO, base, [gold, test_patch], tests)
        f2p = [t for t in f2p if again.get(t)]
        p2p = [t for t in p2p if again.get(t)]
    if not f2p:
        return None

    issue = pr["closingIssuesReferences"]
    task_text = pr["title"] + "\n\n" + (pr["body"] or "")
    return {
        "instance_id": f"{GH_REPO.replace('/', '__')}-{pr['number']}",
        "base_commit": base,
        "problem_statement": scrub(task_text),   # remove code the PR itself added!
        "linked_issues": [i["number"] for i in issue],
        "patch": gold, "test_patch": test_patch, "test_files": tests,
        "FAIL_TO_PASS": f2p, "PASS_TO_PASS": p2p,
    }

def scrub(text: str) -> str:
    """PR descriptions often describe HOW the fix works. Strip code blocks and
    diff snippets; better still, prefer the linked issue's text, or have a
    human rewrite the task as a user would have reported it."""
    text = re.sub(r"```.*?```", "[code removed]", text, flags=re.S)
    return re.sub(r"^[+-].*$", "", text, flags=re.M)

if __name__ == "__main__":
    with open("my_bench.jsonl", "w") as f:
        for pr in merged_prs():
            inst = build_instance(pr)
            if inst:
                f.write(json.dumps(inst) + "\n")

A practical recipe for teams

1. Start with 30 to 50 hand-reviewed tasks. That is enough to catch big regressions and to compare harness choices.
2. Run each agent configuration 3 or more times per task: agents are stochastic, and you want pass@1 with a confidence interval, and ideally pass^3.
3. Record the full scorecard (resolved, regressions, diff size, judge scores, cost, turns) and store every trajectory.
4. Add new tasks every month from freshly merged PRs. They are, by construction, after most models' training data, which gives you a private, contamination-resistant slice for free.
5. Wire the suite into CI for your agent's prompt and tool changes (Module 15).

Code-review agents

A growing class of coding agents does not write the fix at all: it reviews a pull request and leaves comments. Evaluating review is harder than evaluating fixes because there is no test to run. The usual building blocks:

• Seeded-bug recall. Take known-good PRs, inject a bug (or use historical PRs that were later reverted or hot-fixed), and measure how often the reviewer flags the right location. This gives recall: of the real problems, what fraction did it catch?
• Comment precision. Have humans or a calibrated LLM judge label each comment as valid-and-useful, valid-but-trivial (nitpick), or wrong. Precision matters enormously here: a reviewer that raises ten false alarms per PR gets ignored, no matter its recall.
• Acceptance in production. Once deployed, track the rate at which developers resolve, apply or dismiss the agent's comments. This is an online eval signal (Module 15), noisy but grounded in real behavior.
• Matching comments to ground truth. A comment "counts" if it points at the right file and line range and describes the same problem. Line matching is code; "same problem" usually needs a judge.

Report precision and recall together (or an F-score), per severity level, and look at examples. A reviewer tuned only for recall becomes a noise machine.

Evaluating coding assistants as products

Everything so far asks one question: can this model and harness solve a coding task? A company that ships a coding assistant (an autocomplete like Copilot or Cursor Tab, an in-editor agent, or a terminal agent like Claude Code) has a different question: does this product make real developers ship better software, faster, on their own code, week after week? A benchmark cannot answer that, for the same reason a car's lab emissions test cannot tell you how people actually drive it. You need measurements taken from real use.

The product metrics funnel

Product teams think of assistant output as flowing through a funnel. Each step down is a stricter answer to "did the assistant help?":

• Shown (impressions). How many suggestions, proposed edits or agent diffs the developer actually saw. This is the denominator, and the product itself controls it, which matters a lot below.
• Accepted. The developer pressed Tab, clicked apply, or kept the agent's change. Acceptance rate = accepted / shown. Teams also track accepted characters or lines, because accepting a one-character suggestion and a 40-line function are very different events.
• Survived (edit survival, also called code retention or persistence). Of the text that was accepted, how much is still there after N minutes, hours or days? Accepting a suggestion and then deleting it thirty seconds later is not help. Survival is usually computed by comparing the accepted span with the file at a later time.
• Merged. Did the change reach a merged pull request? For agent products the natural unit is the task or PR: PR merge rate for agent-authored PRs, time to merge (open to merged), and review rounds before merge.
• Stayed good. No revert, no hot-fix to the same lines, no incident traced back to it, within some window after merge. This is the closest you get to "the code was actually correct".

Published work gives these metrics some grounding. GitHub researchers (Ziegler et al., 2022) compared usage measures from Copilot telemetry with developers' own survey ratings of productivity, and the paper reports that the acceptance rate of shown suggestions predicted perceived productivity better than more detailed measures such as how much accepted code persisted. Google's research blog posts on its internal ML code completion report acceptance rate and the fraction of new code characters that came from accepted suggestions as headline metrics. GitHub's Copilot usage metrics for organizations expose similar counts (suggestions shown and accepted, lines accepted, active users). So acceptance-style metrics are real, widely used and cheap. They are also the metric most easily fooled.

Why acceptance rate alone misleads

Acceptance rate is a ratio, and the product controls the denominator. If a new version shows suggestions only when it is very confident, it shows fewer of them, and the ones it shows are accepted more often. The rate goes up even if developers receive less help in total. Cursor's post on training its Tab model with online reinforcement learning reports exactly this shape of change, with fewer suggestions shown and a higher accept rate, which is a good outcome if total useful code held up. You only know that by looking at more than the rate.

The other traps, each with the metric that guards against it:

For agentic products the unit shifts from the keystroke to the task. Suggestion-level metrics fade and teams watch session-level ones instead: did the session end in an accepted change, how many times did the human have to step in or correct course, how many permission prompts and turns did it take, what did it cost in tokens per merged change, how many review comments do agent-authored PRs attract, and how often are they reverted. These map directly onto the offline scorecard from the "Beyond pass/fail" section above, which is the point: the same questions, asked of real use.

Measuring edit survival yourself

If you run an assistant inside your company, you can compute survival from two things you already have: a log of what was accepted (file, inserted text, time) and git. The idea: for each accepted span, look at the same file N days later and count how many of its lines are still there. Line-level matching is crude, but it is honest and easy to explain.

"""Edit survival: what share of accepted assistant-written lines is still in
the file N days later. A sketch; a real product logs a stable suggestion id and
the exact inserted span at accept time."""
import subprocess
from collections import Counter
from datetime import datetime, timedelta

MIN_LEN = 8   # ignore lines like "}" or "else:" that survive by accident

def git(repo: str, *args: str) -> str:
    return subprocess.run(["git", "-C", repo, *args], capture_output=True,
                          text=True, check=True).stdout

def rev_at(repo: str, when: datetime, branch: str = "main") -> str:
    """Last commit on the branch at or before a moment in time."""
    return git(repo, "rev-list", "-1", f"--before={when.isoformat()}", branch).strip()

def file_at(repo: str, path: str, rev: str) -> str:
    try:
        return git(repo, "show", f"{rev}:{path}")
    except subprocess.CalledProcessError:
        return ""                      # file deleted: nothing survived

def surviving(inserted: str, later_file: str) -> tuple[int, int]:
    wanted = [l.strip() for l in inserted.splitlines() if len(l.strip()) >= MIN_LEN]
    pool = Counter(l.strip() for l in later_file.splitlines())
    kept = 0
    for line in wanted:                # multiset match: each later line used once
        if pool[line] > 0:
            pool[line] -= 1
            kept += 1
    return kept, len(wanted)

def survival_rate(events: list[dict], repo: str, days: int = 30) -> float:
    """events: [{"path": "src/app.py", "inserted": "...", "accepted_at": datetime}]"""
    kept = total = 0
    for e in events:
        rev = rev_at(repo, e["accepted_at"] + timedelta(days=days))
        k, n = surviving(e["inserted"], file_at(repo, e["path"], rev))
        kept, total = kept + k, total + n
    return kept / total if total else float("nan")

Three honest caveats. First, a line that was rewritten slightly (a renamed variable) counts as lost, so survival is a lower bound. Second, code in files nobody touches again "survives" by default; compare against a baseline of human-written lines in the same files over the same window, otherwise you are measuring how busy the file is. Third, a renamed or moved file looks deleted unless you follow renames (git log --follow). Code-analytics vendors such as GitClear have published repository-level analyses arguing that short-lived and duplicated code rose as AI assistants spread. Those studies are observational (they cannot separate the tool from everything else that changed), so treat them as a hypothesis to check in your own history, not as a measured effect.

Online A/B tests for coding assistants

To know whether version B of the assistant is better than version A in real use, run a randomized experiment, exactly as in Module 15, with a few coding-specific rules:

• Randomize by developer, not by request. A developer who sees A in the morning and B in the afternoon learns habits from both, and a session that switches versions mid-task is contaminated.
• Pick one primary metric in advance (for example surviving characters per developer per day) and list guardrail metrics that must not get worse: latency, error rate, revert rate, cost per developer.
• Analyze at the developer level. One heavy user produces thousands of correlated suggestions. If you treat every suggestion as independent, your confidence interval will be far too narrow.
• Run long enough to see survival and merges, which lag acceptance by days or weeks, and long enough for novelty to wear off.

import numpy as np

rng = np.random.default_rng(0)

def dev_bootstrap_ci(shown_a, acc_a, shown_b, acc_b, n_boot=5000):
    """95% CI for (rate_B - rate_A), resampling DEVELOPERS, not suggestions.
    Each array has one entry per developer: suggestions shown / accepted."""
    shown_a, acc_a = np.asarray(shown_a), np.asarray(acc_a)
    shown_b, acc_b = np.asarray(shown_b), np.asarray(acc_b)
    diffs = np.empty(n_boot)
    for i in range(n_boot):
        ia = rng.integers(0, len(shown_a), len(shown_a))   # resample developers
        ib = rng.integers(0, len(shown_b), len(shown_b))
        diffs[i] = acc_b[ib].sum() / shown_b[ib].sum() - acc_a[ia].sum() / shown_a[ia].sum()
    return np.percentile(diffs, [2.5, 97.5])

Developer-reported productivity, and why it misleads

The cheapest way to ask "does this help?" is to ask developers. Surveys are genuinely useful for finding friction ("it keeps suggesting the old API") and for measuring satisfaction. They are a poor measure of speed. The clearest evidence is a randomized controlled trial by METR, published in July 2025. Experienced open-source developers worked on real issues from large, mature repositories they already maintained; each issue was randomly assigned to "AI tools allowed" or "AI tools not allowed", and the time to finish was recorded. The paper reports that developers forecast before starting that AI would cut their completion time by about 24%, estimated afterwards that it had cut it by about 20%, and in fact took about 19% longer when AI was allowed.

Read this result carefully, in both directions. It does not show that AI tools slow everyone down: the study was small (the paper reports 16 developers and 246 tasks), the developers were experts in codebases they knew deeply, and the tools were those of early 2025. METR itself lists these limits, and in February 2026 it published a note explaining that it was changing its experiment design for follow-up studies. What it does show, very clearly, is that self-reports can point in the opposite direction from measured time. Reasons include: waiting on the model feels like work but is idle time, reviewing and fixing generated code is easy to forget, novelty feels productive, and nobody wants to seem slow to adopt new tools.

Putting it together: a product eval program for a coding assistant

1. Offline gate. A repo-history suite (previous section) plus a few public benchmarks, run on every model, prompt or harness change. Nothing ships that regresses here.
2. Replay set. With consent, keep a stream of anonymized real requests (the prompt, the repository state, what the developer finally committed). Re-run candidates on them offline and compare their output with what the developer actually shipped, using tests where they exist and a calibrated judge otherwise (Module 13).
3. Dogfood and canary. Internal users first, then a small percentage of real users, watching guardrail metrics (latency, errors, reverts).
4. A/B test for anything big, randomized by developer, with one pre-registered primary metric from low in the funnel (surviving code, merged and not reverted) and developer-level confidence intervals.
5. Qualitative loop. Read sessions where the developer rejected or rewrote the output (Module 12). Each new failure mode becomes an offline task.

Check yourself

Sources and further reading

• Chen et al. (OpenAI), Evaluating Large Language Models Trained on Code (HumanEval, pass@k), 2021.
• Austin et al. (Google), Program Synthesis with Large Language Models (MBPP), 2021.
• Liu et al., Is Your Code Generated by ChatGPT Really Correct? (EvalPlus, HumanEval+ / MBPP+), 2023.
• Jimenez, Yang et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, ICLR 2024; site and leaderboards swebench.com.
• OpenAI, Introducing SWE-bench Verified, Aug 2024.
• OpenAI, Why SWE-bench Verified no longer measures frontier coding capabilities, Feb 2026.
• SWE-bench GitHub, Issue #465: Repo state loopholes during agentic evaluation (git history leakage) and Issue #669: solutions reachable from problem_statement.
• UTBoost write-up, SWE-bench Verified is flawed despite expert review.
• Yang et al., SWE-bench Multimodal, 2024.
• Scale AI, SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?, Sep 2025; public leaderboard.
• Yang et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, 2024.
• Anthropic Engineering, Raising the bar on SWE-bench Verified with Claude 3.5 Sonnet, Jan 2025.
• Merrill, Shaw, Carlini, Schmidt et al., Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces, Jan 2026; tbench.ai.
• Harbor docs, Task structure and Harbor repository.
• Aider, Aider LLM leaderboards (polyglot benchmark).
• Jain et al., LiveCodeBench, 2024; leaderboard.
• Chan et al. (OpenAI), MLE-bench, 2024.
• Wijk et al. (METR), RE-Bench, 2024.
• Starace et al. (OpenAI), PaperBench, 2025.
• Miserendino et al. (OpenAI), SWE-Lancer, 2025.
• SWE-rebench, swe-rebench.com (continuously refreshed, decontaminated tasks).
• Becker, Rush et al. (METR), Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, July 2025; METR blog summary; We are Changing our Developer Productivity Experiment Design, Feb 2026.
• Ziegler et al. (GitHub), Productivity Assessment of Neural Code Completion, 2022; later version in CACM, Measuring GitHub Copilot's Impact on Productivity, 2024.
• Google Research, ML-Enhanced Code Completion Improves Developer Productivity and AI in software engineering at Google: Progress and the path ahead.
• GitHub Docs, GitHub Copilot usage metrics.
• Cursor, Improving Cursor Tab with online RL.
• GitClear, AI Copilot Code Quality: 2025 research (vendor, observational).
• Anthropic, How AI is transforming work at Anthropic.