Evaluating the agent harness

An agent is a model wrapped in code: a system prompt, tools, a loop, rules for managing context, notes, sub-agents, permissions and checks. That wrapper is called the harness, and it can move a score as much as a model upgrade. This module shows how to evaluate the harness on its own terms: hold the model fixed, change one thing, measure fairly, and find out which parts of your harness actually earn their keep.

Two things called "harness"

The word "harness" is used for two different pieces of software in agent evaluation, and mixing them up causes real confusion in papers, docs and interviews. Anthropic's engineering post Demystifying evals for AI agents (January 2026) draws the line clearly:

• The agent harness (also called the scaffold) is the code that turns a model into an agent: the loop that calls the model, runs the tools it asks for, feeds results back, manages the context, and decides when to stop. It is part of your product.
• The evaluation harness is the test rig: the infrastructure that loads tasks, resets environments, runs many trials, records transcripts, calls the graders and computes metrics. Inspect, Harbor and HAL are evaluation harnesses. It is part of your measuring instrument.

The same post makes a point worth memorizing: when you evaluate "an agent", you are evaluating the harness and the model together. There is no such thing as the score of the model alone once tools and a loop are involved.

Throughout this module, "harness" on its own means the agent harness. When we mean the test rig, we will say "evaluation harness".

Anatomy of an agent harness

Before you can evaluate a harness you need a parts list. Different frameworks use different names, but almost every production agent harness has these eight parts. Each is a knob someone can turn, and each is therefore something you can ablate (switch off) and measure.

Two terms from that table deserve a proper definition. The agent-computer interface (ACI) is the SWE-agent paper's name (Yang et al., 2024) for the set of commands and feedback formats an agent uses to act, designed for a model the way a user interface is designed for a person. Compaction means replacing older parts of the conversation with a shorter summary so the run can continue inside the context window. Anthropic's Effective context engineering for AI agents (September 2025) names compaction, structured note-taking and sub-agent architectures as the three main techniques for long tasks, which is why all three appear in our parts list.

Why the harness deserves its own evals

Module 05 and Module 07 already collected the headline evidence: the SWE-agent ACI results, Anthropic's two-tool SWE-bench scaffold, the mini-SWE-agent bash-only view, and Terminal-Bench reporting model × harness pairs. Here are three further reasons, each of which changes how you run evals.

1. Leaderboards now rank agent + model pairs

The Terminal-Bench 2.0 leaderboard ranks agent + model pairs, and its official evaluation harness, Harbor, can run Claude Code, OpenHands, Codex CLI and other agents on exactly the same containerized tasks. So the same model appears several times with different harnesses, and the same harness appears with different models. The unit of measurement is the pair. (The benchmark is described in Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces.)

2. Standardized harness evaluation at scale: HAL

The Holistic Agent Leaderboard (HAL; Kapoor et al., Princeton; arXiv 2510.11977, accepted at ICLR 2026; site at hal.cs.princeton.edu) is an evaluation harness built to answer the harness question systematically. It runs many agent scaffolds × models × benchmarks under one standardized setup and reports cost next to accuracy for every cell. The HAL paper reports roughly 21,730 rollouts across 9 models and 9 benchmarks at a cost of about $40,000. Among its findings, the paper reports that higher reasoning effort reduced accuracy in the majority of runs, and that LLM-aided inspection of the logs uncovered behaviors no score would show, such as agents searching online for the benchmark answers instead of solving the task, and an agent misusing a credit card in a flight-booking task.

3. Most of what you change day to day is harness

If you build an agent product, you rarely change the model: you change it a few times a year when a new one ships. You change the harness every week. Someone rewrites a paragraph of the system prompt, adds a tool, raises the turn limit, lowers the compaction threshold, or turns on a sub-agent. Each of those edits is a release of your agent that can make things better or quietly worse. Anthropic's Writing effective tools for agents (September 2025) recommends exactly this discipline for tool changes: build realistic evaluation tasks, run them programmatically, track accuracy alongside runtime, number of tool calls, token use and tool errors, read the transcripts, and keep a held-out set so you do not overfit the tool descriptions to your test tasks.

The core method: hold the model fixed, change one thing

Every harness eval is a small controlled experiment. The whole method fits in one sentence: keep everything identical except the one harness component you are testing, run both versions on the same tasks several times, and compare them task by task. Each word in that sentence prevents a specific mistake.

The rules, and what each one prevents

A worked example of the paired interval

Suppose you remove compaction from your harness and run 150 tasks, 3 epochs each, with both configs. For each task you average its 3 epochs, then subtract: d_i = (success rate without compaction) − (success rate with compaction). You get 150 numbers, most of them 0 (the task passes or fails either way), some negative, a few positive.

• Mean of the 150 differences: −0.067 (a drop of 6.7 points).
• Standard deviation of the differences: 0.19.
• Standard error: 0.19 / √150 = 0.19 / 12.25 ≈ 0.0155.
• 95% interval: −0.067 ± 1.96 × 0.0155 = −0.067 ± 0.030, so [−9.7, −3.7] points.

The interval excludes zero, so compaction genuinely helps on this task set. Had you compared the two configs as independent samples, the standard error would come from each config's own spread (around 0.4 for per-task rates near 60%), giving an interval of roughly ±9 points that straddles zero. Pairing is not a nicety: it is the difference between a conclusion and a shrug. These numbers are illustrative.

Here is the whole comparison in code. It reads two Inspect logs (one per config), pairs tasks by id, and bootstraps the mean difference over tasks, carrying each task's epochs together as Module 04 recommends.

# paired_ablation.py : compare two harness configs task by task
import glob, random
from collections import defaultdict
from inspect_ai.log import read_eval_log

def per_task_rates(log_dir: str, scorer: str = "hidden_tests") -> dict:
    """Average the epochs of each task: one success rate per task id."""
    path = sorted(glob.glob(f"{log_dir}/*.eval"))[-1]      # latest run in that folder
    log = read_eval_log(path)
    runs = defaultdict(list)
    for s in log.samples:                                  # one entry per task x epoch
        runs[s.id].append(1.0 if s.scores[scorer].value == "C" else 0.0)
    return {tid: sum(v) / len(v) for tid, v in runs.items()}

def paired_diff(a: dict, b: dict, n_boot: int = 10_000, seed: int = 0):
    """Mean of (b - a) over shared tasks, with a percentile bootstrap 95% CI."""
    ids = sorted(set(a) & set(b))                          # pair by task id
    d = [b[i] - a[i] for i in ids]
    mean = sum(d) / len(d)
    rng = random.Random(seed)
    boots = sorted(sum(rng.choice(d) for _ in d) / len(d) for _ in range(n_boot))
    return mean, boots[int(0.025 * n_boot)], boots[int(0.975 * n_boot)], len(ids)

base = per_task_rates("logs/full")
cand = per_task_rates("logs/no_compaction")
m, lo, hi, n = paired_diff(base, cand)
print(f"n={n} tasks  delta={m*100:+.1f} pts  95% CI [{lo*100:+.1f}, {hi*100:+.1f}]")
# n=150 tasks  delta=-6.7 pts  95% CI [-9.7, -3.7]     (illustrative output)

Ablation studies: which parts earn their keep?

An ablation (a word borrowed from surgery and neuroscience, where it means removing tissue to see what stops working) switches off one component at a time and measures the damage. Run the full harness, then the full harness minus X, minus Y, and so on. The resulting ablation table is the single most useful artifact in harness engineering: it tells you which components are load-bearing, which are dead weight, and what each one costs. The SWE-agent paper is a good published example: alongside its headline number it reports ablations of individual interface components, such as its file viewer, search commands and the linter that rejects syntactically broken edits.

A worked ablation table

Here is an illustrative ablation for a coding agent, 150 tasks × 3 epochs, one model, identical budgets (80 turns, 400k tokens per trial). The full harness scores 62.0%.

Try it: switch harness components off and watch success rate and cost move.

How to read an ablation table like an expert

• "No significant drop" is not "useless". The planning-file interval is [−3.0, +1.2]: the data is consistent with the component helping by up to 3 points. You can say "no evidence it helps on this suite", not "it does nothing". A bigger suite, or a suite of longer tasks, may reveal an effect.
• Accuracy and cost are separate verdicts. Truncating tool output barely moves success but saves a lot of tokens, so you keep it. Sub-agents help, but at a price; whether that is worth it is a product decision, not a statistical one. A cost-per-success column (tokens divided by success rate) makes the trade-off concrete.
• Leave-one-out and add-one-in answer different questions. Removing X from the full harness measures X's contribution given everything else. Adding X to a minimal harness measures its contribution alone. When components overlap (verification and retries both catch errors), each looks unimportant in leave-one-out because the other covers for it, yet removing both is disastrous. When you suspect overlap, also ablate the pair.
• Effects are task-dependent. Compaction only matters for tasks long enough to fill the context. Slice the table by task length or type before concluding. An average over a suite of mostly short tasks can hide a component that is essential for the long ones.
• Look at the component metrics, not just success. If removing retries drops success, the tool error rate and exit reasons should show why. If the success number moves but the mechanism does not, suspect noise or a confound.

The model × harness matrix

An ablation holds the model fixed. The next question is what happens when you vary both: several models, several harnesses, every combination. The result is a model × harness matrix, and it reveals something a single-model ablation cannot: interaction effects. An interaction means the effect of the harness depends on which model is inside it. Prompts, tool formats and workflow hints are often written around one model's habits (how verbose it is, how it formats tool calls, whether it tends to stop early). Swap the model and those same choices can hurt.

What to report from a matrix

• Marginal means (row and column averages) answer "which harness is best on average across models?" and "which model is best on average across harnesses?". They are the robust answer when you expect to swap models.
• Per-cell results answer "which pair should I ship?". The best pair is not always the best model in the best-on-average harness: here the best cell is A + H2 (66), even though H3 has the highest column mean.
• Rank flips are the headline. In H1, B beats A (55 vs 48). In H2, A beats B (66 vs 52). If you had compared models only in H2, you would have concluded A is much stronger. This is why SWE-bench added a bash-only view with one shared minimal harness for model comparisons, as Module 07 describes.
• Uncertainty per cell. Nine cells means nine intervals and many comparisons. Run enough tasks and epochs that the differences you care about exceed the noise, and correct for multiple comparisons or confirm the winner on held-out tasks (Module 04).

Component-level metrics: seeing inside the run

Success rate tells you whether a harness change helped. Component-level metrics tell you why, and they catch regressions that the success rate hides (a harness can keep the same success rate while doubling its tool errors and recovering from them, which costs money and will eventually break). You compute them from the transcript: the full log of every message, tool call, tool result and harness event in a trial.

The metrics, defined

Two of these deserve a closer look because they are both important and easy to get wrong.

Premature "done": when the agent's own report is wrong

An agent that says "all done" when it is not is worse than one that says "I could not finish", because a human or a calling program will trust the claim. Anthropic's post on long-running harnesses names "declaring the job done too early" as one of the main failure modes that its harness design targets. To measure it you need two independent signals per trial: what the agent claimed, and what the end-state grader found.

State retention after compaction

Compaction replaces old turns with a summary. If the summary drops a detail the agent needs later, the failure shows up many turns afterwards and looks unrelated. The fix is to probe right after the event: the harness logs each compaction, and your evaluation harness then asks the agent a question whose answer the environment knows ("which tests are still failing?", "which orders are still pending?"). Module 11 shows the sawtooth picture of context filling and compacting, and explains probing; for harness evals, the key move is to compare probe accuracy between compaction strategies (for example, summary only vs summary plus a notes file) under the same tasks. Keep probes out of the agent's real context if you can (fork the state, ask, discard the answer), so the probe itself does not act as a helpful reminder that changes the run.

A compaction test suite, step by step

Compaction is the harness component most likely to fail silently, so it deserves a dedicated suite rather than a hope that end-to-end tasks will expose it.

1. Plant facts early. Write tasks where a detail appears early and is needed late: a constraint in the first user message ("never touch the billing/ folder"), an identifier returned by an early tool call, a decision the agent made and must stay consistent with.
2. Force the event. Pad the middle of the run with realistic but bulky work (long logs, large files) so that compaction reliably fires before the fact is needed. Log the turn and the token counts before and after each compaction.
3. Probe and act. Ask a forked probe question right after the event, and also check whether the agent's later actions respect the planted fact. An agent can answer the probe correctly and still violate the constraint.
4. Compare strategies on identical runs. No compaction (where it fits), summary only, summary plus a notes file, clearing old tool results only. Same tasks, same epochs, paired.
5. Sweep the threshold. Compacting at 50%, 70% or 90% of the window trades lost detail against context rot. Plot retention and success against the threshold, with cost.
6. Grade the summary itself. Keep a list of the facts each task needs; check which ones survive in the summary text (a code check or a validated judge, Module 13). This tells you which kinds of facts your compaction prompt drops: numbers, identifiers and negative constraints ("do not ...") are common casualties.

Computing the metrics in code

Most harnesses can emit a structured event log. The snippet below assumes one list of events per trial and computes every metric above. It is plain Python, so it works with any framework: map your framework's transcript format onto these event types first.

# harness_metrics.py : component-level metrics from transcript events
import json

# One trial = an ordered list of events your harness logged, e.g.
# {"turn": 3, "type": "tool_call", "tool": "edit", "args": {"line": 12}, "valid_args": True}
# {"turn": 3, "type": "tool_result", "error": None}
# {"turn": 3, "type": "context", "tokens": 18250}           # context size before this turn
# {"turn": 7, "type": "compaction", "before": 38000, "after": 6000}
# {"turn": 8, "type": "probe", "after_compaction": True, "correct": False}
# {"turn": 9, "type": "handoff", "brief_fields": ["goal"]}
# {"turn": 10, "type": "final", "claims_done": True}
# ...and after the run, the grader appends {"type": "end_state", "passed": False}

REQUIRED_BRIEF = {"goal", "files", "failing_tests", "done_criteria"}

def trial_metrics(events: list, loop_window: int = 3) -> dict:
    of = lambda t: [e for e in events if e["type"] == t]
    calls, results, probes, handoffs = of("tool_call"), of("tool_result"), of("probe"), of("handoff")
    ctx = [e["tokens"] for e in of("context")]
    final = next(iter(of("final")), {"claims_done": False})
    end = of("end_state")[0]

    # 1. Tool errors and invalid arguments, per tool call
    n = max(len(calls), 1)
    tool_error_rate = sum(r["error"] is not None for r in results) / n
    invalid_arg_rate = sum(not c["valid_args"] for c in calls) / n

    # 2. Loops: longest run of identical (tool, args) calls in a row
    sigs = [(c["tool"], json.dumps(c["args"], sort_keys=True)) for c in calls]
    longest, run = (1 if sigs else 0), 1
    for prev, cur in zip(sigs, sigs[1:]):
        run = run + 1 if cur == prev else 1
        longest = max(longest, run)

    # 3. Context growth: average tokens added per turn
    growth = (ctx[-1] - ctx[0]) / (len(ctx) - 1) if len(ctx) > 1 else 0.0

    # 4. State retention: probes asked right after a compaction event
    post = [p for p in probes if p.get("after_compaction")]
    retention = sum(p["correct"] for p in post) / len(post) if post else None

    # 5. Handoff quality: share of required brief fields actually present
    handoff = (sum(len(REQUIRED_BRIEF & set(h["brief_fields"])) / len(REQUIRED_BRIEF)
                   for h in handoffs) / len(handoffs)) if handoffs else None

    return {
        "success": end["passed"],
        "steps": len(calls),
        "tool_error_rate": tool_error_rate,
        "invalid_arg_rate": invalid_arg_rate,
        "stuck": longest >= loop_window,
        "tokens_per_turn": growth,
        "retention_after_compaction": retention,
        "handoff_quality": handoff,
        "claims_done": final["claims_done"],
        "premature_done": final["claims_done"] and not end["passed"],
    }

def suite_summary(trials: list) -> dict:
    rows = [trial_metrics(t) for t in trials]
    mean = lambda xs: sum(xs) / len(xs) if xs else float("nan")
    wins = [r for r in rows if r["success"]]
    claims = [r for r in rows if r["claims_done"]]
    return {
        "success_rate": mean([r["success"] for r in rows]),
        "steps_to_success": mean([r["steps"] for r in wins]),            # successes only
        "tool_error_rate": mean([r["tool_error_rate"] for r in rows]),
        "stuck_rate": mean([r["stuck"] for r in rows]),
        "premature_done_rate": mean([r["premature_done"] for r in claims]),  # of "done" claims
        "retention_after_compaction": mean([r["retention_after_compaction"] for r in rows
                                            if r["retention_after_compaction"] is not None]),
        "mean_handoff_quality": mean([r["handoff_quality"] for r in rows
                                      if r["handoff_quality"] is not None]),
    }

Notice two deliberate choices. Steps to success is averaged over successful trials only; averaging over all trials mixes "fast success" with "gave up early", which move in opposite directions. Premature-done rate is divided by the number of "done" claims, not by all trials, so it measures how trustworthy a claim is. Report these per config, next to the success rate and its interval, in every harness comparison.

Running harness variants side by side

In practice, you express each harness variant as a configuration of your evaluation harness, so that everything else stays byte-for-byte identical. Two common setups:

Your own harness, in Inspect

If your agent is built in (or bridged into) Inspect, make the harness variant a task parameter. The dataset, scorer, sandbox, limits and epochs are defined once and shared, so the only thing a parameter can change is the harness. The pattern below follows the task and react() agent examples in the Inspect docs and in Module 02; check parameter names against your installed version.

# harness_ab.py : one task definition, several harness variants
from inspect_ai import Epochs, Task, task
from inspect_ai.agent import react
from inspect_ai.dataset import json_dataset
from inspect_ai.tool import bash, text_editor

from my_graders import hidden_tests          # the SAME outcome grader for every variant

PROMPT_V1 = open("prompts/system_v1.txt").read()
PROMPT_V2 = open("prompts/system_v2.txt").read()   # adds "run the tests before you finish"

def build_agent(harness: str):
    """Each variant differs from 'baseline' in exactly one component."""
    if harness == "baseline":
        return react(prompt=PROMPT_V1, tools=[bash(timeout=120), text_editor()])
    if harness == "verify_prompt":                  # change: system prompt only
        return react(prompt=PROMPT_V2, tools=[bash(timeout=120), text_editor()])
    if harness == "bash_only":                      # change: tool set only
        return react(prompt=PROMPT_V1, tools=[bash(timeout=120)])
    raise ValueError(f"unknown harness {harness!r}")

@task
def repo_tasks(harness: str = "baseline", epochs: int = 3):
    return Task(
        dataset=json_dataset("tasks.jsonl"),        # SAME tasks
        solver=build_agent(harness),
        scorer=hidden_tests(),                      # SAME grader
        sandbox="docker",                           # SAME environment image
        message_limit=80,                           # SAME budget: turns ...
        token_limit=400_000,                        # ... and tokens
        # mean = success rate; at_least_3 = passed all 3 epochs (a per-task pass^3)
        epochs=Epochs(epochs, ["mean", f"at_least_{epochs}"]),
    )

# Same model, same flags; only the harness parameter changes (-T sets task arguments)
for h in baseline verify_prompt bash_only; do
  inspect eval harness_ab.py -T harness=$h \
      --model anthropic/claude-sonnet-5-5 --log-dir logs/$h
done
python paired_ablation.py      # pair each variant with logs/baseline, as shown earlier

Off-the-shelf agents, in Harbor

If the harnesses you are comparing are complete CLI agents (Claude Code, OpenHands, Codex CLI and so on), you cannot easily edit their internals, but you can hold the model and the tasks fixed and swap the agent. That is exactly what Terminal-Bench's harness, Harbor, is built for:

# Same model and task set; the agent harness is the only thing that changes
harbor run --dataset terminal-bench@2.0 --agent claude-code \
           --model anthropic/claude-opus-5-5 --n-concurrent 4
harbor run --dataset terminal-bench@2.0 --agent <another-supported-agent> \
           --model anthropic/claude-opus-5-5 --n-concurrent 4
# (see the Harbor README for the list of supported agent names)

This gives you whole-harness comparisons (which product is better with this model?) rather than component ablations. Both are useful: whole-harness comparisons tell you where you stand, component ablations tell you what to change.

Long-running harnesses: evaluating across context windows

Some agents work for hours and many context windows. No single window sees the whole job, so the harness must carry state from one window to the next through files. Anthropic's Effective harnesses for long-running agents (November 2025) describes a two-part pattern for coding:

• An initializer agent runs first. It sets up the environment: a feature list file that spells out what "done" means feature by feature, a progress notes file, and a git repository, so later sessions have a map and a history.
• A coding agent then runs repeatedly, each time in a fresh context window. It reads the progress notes and the git log to get its bearings, works on one feature at a time, tests it end to end (the post uses browser automation for this), commits, and updates the notes for the next session.

The post frames the design around three failure modes: trying to do everything at once, declaring the job done too early, and leaving the environment in a broken state for the next session. Each of these is something you can measure.

What to check in a long-running harness

Treat every session boundary as a checkpoint where your evaluation harness inspects the environment (it can do this outside the agent's view):

The last row is the most informative single chart for a long-running harness: verified features passing versus session number, for each harness variant, with the agent's own claimed count overlaid. A good harness climbs steadily and the two lines agree. A poor one plateaus, zig-zags as later sessions break earlier work, or shows a claimed line running above the verified one. Module 11 covers milestone scoring and partial credit for long tasks, which you can reuse here.

Harnesses with a built-in evaluator agent

A follow-up Anthropic post, Harness design for long-running application development, describes using a multi-agent harness to push Claude further in frontend design and long-running autonomous software engineering; write-ups of the post describe its roles as a planner, a generator that builds, and a separate evaluator that critiques the generator's output. A harness like this contains its own grader, which raises a new evaluation question: is the in-harness evaluator any good? Treat it exactly like an LLM judge (Module 13). Sample its verdicts, have humans (or your independent end-state checks) grade the same outputs, and measure agreement, false passes and false fails. Then ablate it: generator alone, generator plus evaluator, at matched budgets. An in-harness evaluator that approves broken work is a premature-done machine, and one that rejects good work burns budget; both show up only if you grade the evaluator separately from the final outcome. Never reuse the harness's own evaluator as your eval's grader: the instrument must be independent of the thing it measures.

Is it the model or the harness?

When a task fails, the first question in a harness team is always whose fault it was. Here is a diagnostic sequence that works on a single failing task or a cluster of them. It goes from cheapest to most expensive check.

Localizing failures on the interactions

The triage tree asks "model or harness?" as if the answer were one box. Often the fault sits on the edge between two boxes: the model misreads a tool's output format, the harness truncates exactly the part of a result the model needed, a sub-agent's brief omits what the parent knew. A 2026 paper, Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures, takes this view: it organizes agent failure modes by the interaction they originate in, assigning each to a pair of components such as model and harness, model and tools, or harness and memory. For your own error analysis (Module 12), the practical lesson is to code each failure with two labels, the component that produced the bad information and the component that consumed it, rather than a single "model" or "harness" tag. Counting failures per edge points you straight at the interface to fix, and it stops the familiar argument in which the harness team blames the model and the model team blames the harness.

Check 2 is the most underused. When an agent makes a wrong decision, find the step and look at exactly what was in its context window at that moment. If the needed fact was truncated, compacted away or never retrieved, the model never had a chance, and no model upgrade will fix it reliably. If the fact was right there and the model still ignored it, look at prompt structure and context length (a fact buried in 150k tokens of tool output is technically "in context" but practically hidden), and only then blame the model.

Harness regression testing in CI

Because the harness changes far more often than the model, harness evals belong in continuous integration (CI). The rule is simple: every harness change is a release. A one-line prompt edit, a new tool, a changed tool description, a new compaction threshold, a higher turn cap: each one reruns the suite and must pass a gate before it merges. Module 15 shows how to wire evals into CI in general; here is what is specific to harness changes.

What to gate on, and why

• Success, as a paired interval. Gate on the lower bound of the paired difference ("we are confident it did not drop by more than 2 points"), not on the point estimate. A point-estimate gate fails randomly half the time on a change that does nothing.
• pass^k on a golden set. pass^k (Module 04) is the chance that all k attempts succeed. Harness changes often trade reliability for peak performance: a more exploratory prompt can raise pass@k while lowering pass^k. For an agent that runs unattended, pass^k is the number users feel.
• Cost and latency. A harness change that raises success by 1 point and tokens by 40% is usually a regression. Put a ceiling on cost per task, or gate on cost per success.
• Component metrics. A spike in tool errors, loops or premature "done" claims is a warning even when success holds, because it usually means the agent is recovering by luck.
• Canary tasks. Keep a handful of tasks that exercise each component specifically (a task that only succeeds if compaction preserves a detail; a task that needs a sub-agent), so a component that silently stops working turns a specific canary red.

# harness_gate.py : fail CI if the candidate harness regresses
import sys
from paired_ablation import per_task_rates, paired_diff

MAX_DROP = -0.02          # lower CI bound may not fall below -2 points
MAX_COST_RATIO = 1.10     # at most +10% tokens per task

def gate(base_dir: str, cand_dir: str, base_cost: float, cand_cost: float) -> list:
    problems = []
    m, lo, hi, n = paired_diff(per_task_rates(base_dir), per_task_rates(cand_dir))
    if lo < MAX_DROP:
        problems.append(f"success: delta {m:+.3f}, CI lower bound {lo:+.3f} < {MAX_DROP}")
    if cand_cost > MAX_COST_RATIO * base_cost:
        problems.append(f"cost: {cand_cost:.0f} vs {base_cost:.0f} tokens per task")
    return problems

if __name__ == "__main__":
    issues = gate("logs/baseline", "logs/candidate", base_cost=210_000, cand_cost=224_000)
    for i in issues:
        print("GATE FAIL:", i)
    sys.exit(1 if issues else 0)

Pitfalls

Harness evals fail in characteristic ways. Most of them make a harness look better than it is.

1. Unmatched budgets

The most common mistake. A new harness uses more turns, more tokens, more sub-agents or more parallel attempts, and of course it scores higher. That is not a better harness; it is a bigger budget. The fair comparison is either at matched budgets (same turn and token caps) or on a cost curve: success as a function of budget, for each harness.

2. Overfitting the harness to the benchmark

Every time you look at failures on your eval suite and tweak the prompt to fix them, you spend a little of the suite's ability to tell you the truth. After a few dozen rounds, your prompt contains instructions that address specific tasks ("remember to check the timezone"), and the suite score overstates how the harness will do on new tasks. This is Goodhart's law applied to harness engineering (Module 18). The defenses are the same as in machine learning: a held-out task set you look at rarely, fresh tasks added regularly from production traces, and a rule that prompt instructions must be general (no task-specific hints). Anthropic's tool-writing guidance recommends a held-out set for exactly this reason. The pressure is increasing: a line of 2026 papers explores automated harness optimization, where an optimizer rewrites prompts and tools against an eval, which makes a clean held-out set non-negotiable.

3. Comparing numbers across evaluation harnesses

Your harness scores 58% on a benchmark in your evaluation harness; a blog post reports 64% for another agent harness on "the same" benchmark in theirs. Those numbers are not comparable. Evaluation harnesses differ in timeouts, sandbox images and installed packages, retry-on-infrastructure-error policies, grader versions, task subsets and how they count crashed trials. To compare two agent harnesses, run both inside one evaluation harness. This is the entire reason HAL and Harbor exist.

4. Infrastructure noise: the sandbox is part of the result

Even inside one evaluation harness, the resources each trial gets can move the score. Anthropic's engineering post Quantifying infrastructure noise in agentic coding evals (February 2026) studied exactly this; in announcing it, Anthropic summarized that infrastructure configuration can swing agentic coding benchmarks by several percentage points, sometimes more than the leaderboard gap between top models. The mechanism is easy to picture: with less CPU or memory, builds and test suites run slower or crash, more trials hit their time limit, and an agent that would have succeeded is scored as a failure. A generous machine can also let a brute-force approach finish that would time out elsewhere.

For harness evals this has three consequences. First, pin and record the sandbox resources (CPU, memory, disk, time limits, concurrency) as part of the configuration, exactly like the model version. Second, when comparing harness variants, run them on the same infrastructure, interleaved, never "baseline last month, candidate today on the new cluster". Third, separate infrastructure failures (out-of-memory kills, container start failures, timeouts in setup) from agent failures in your exit reasons, and watch whether a harness change shifts that mix: a harness that spawns sub-agents or runs more tests in parallel can be the thing that pushes trials over a resource limit. Module 18 discusses infrastructure noise as a benchmark pitfall more broadly.

Try it: change the sandbox's resources and time limit and watch how much of the "gap" between two configurations is really infrastructure.

5. Leakage through tools

A harness that gives the agent a web search or unrestricted network access can find the answers instead of solving the task. The HAL paper reports finding exactly this in its logs: agents searching for benchmark answers online. A tool can also leak through the filesystem (the hidden tests are readable in the container) or through git history (a future commit with the fix is present). When a new tool raises the score, read the transcripts of the newly passing tasks before celebrating. Block benchmark domains at the network layer, and keep graders and reference solutions out of the agent's sandbox (Module 18 on reward hacking).

6. The winner's curse over many configurations

Suppose you try 20 prompt variants, none of which is truly better than the baseline, on a suite where the paired standard error of a difference is 2 points. By chance alone, the expected best of 20 noisy differences is about 1.87 standard errors above zero, so the "winner" will look about +3.7 points better, and it is pure noise. This is the winner's curse: the maximum of many noisy estimates is biased upward. The fixes: confirm the winner on a fresh held-out set before shipping, correct for multiple comparisons, and report how many configurations you tried. Leaderboards suffer from the same effect when submitters test many private variants and publish the best (Module 18).

7. Confounding through the environment

Harness A ran on Monday, harness B on Tuesday, and on Tuesday the package mirror was slow, so more of B's trials timed out. Or the model provider updated the model version behind the same alias. Run variants interleaved (alternate or randomize the order of trials across configs), pin model versions, and track infrastructure errors separately from agent failures. Module 05 covers environment determinism.

Putting it together: a harness eval plan

Check yourself

Sources and further reading

• Anthropic, Demystifying evals for AI agents (January 2026): agent harness vs evaluation harness, pass@k vs pass^k.
• Anthropic, Effective harnesses for long-running agents (November 2025): initializer and coding agents, feature list, progress notes, git, end-to-end testing.
• Anthropic, Effective context engineering for AI agents (September 2025): compaction, structured note-taking, sub-agent architectures.
• Anthropic, Writing effective tools for agents (September 2025): evaluation-driven tool development, held-out sets.
• Yang et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (NeurIPS 2024).
• Kapoor et al., Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation (ICLR 2026) and the HAL leaderboard.
• Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (2026), the Terminal-Bench leaderboard, and Harbor.
• UK AI Security Institute and Meridian Labs, Inspect documentation: ReAct agent, agents and Agent Bridge.
• Anthropic, Quantifying infrastructure noise in agentic coding evals (February 2026).
• Anthropic, Harness design for long-running application development (2026).
• Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures (2026).
• Evan Miller, Adding Error Bars to Evals (2024): paired differences and clustered standard errors.
• Recent papers on harness effects, listed for further reading (titles only; their findings are not summarized in this module): Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows; The Scaffold Effect in Coding Agents; An Empirical Study of Harness Design for Coding Agents; Agentic Harness Engineering; HARBOR: Automated Harness Optimization (unrelated to the Terminal-Bench Harbor despite the name).