Multi-agent and long-horizon evals

Two frontiers break the "one prompt, one answer" picture completely: agents that work for hours on a single task, and systems where several agents talk to each other. Both need new yardsticks: time horizons instead of accuracy, milestones instead of pass/fail, and trace trees instead of transcripts.

Two ways an eval gets "big"

Everything so far in this course has had a comfortable shape: one agent, one task, a few dozen steps, a grader at the end. This module is about the two directions in which that shape stretches until it breaks.

They share a common difficulty: the thing you care about (did the whole job get done?) is far away from the things you can easily observe (individual steps, individual messages). The rest of this module is about closing that gap: measuring the whole, and also measuring the parts well enough to know why the whole failed.

Part A · Why long tasks are hard: compounding error

Start with the simplest possible model of an agent. Each step (a tool call, an edit, a decision) succeeds with probability p, independently, and the task needs n steps to all go right. Then the probability of finishing is:

P(task succeeds) = pn

This looks harmless until you plug in numbers. A 99%-reliable step sounds excellent. But over 50 steps, 0.9950 ≈ 0.61, and over 100 steps it is about 0.37. At 95% per step, a 20-step task succeeds only about 36% of the time, and a 100-step task essentially never.

Why the simple model is both too pessimistic and too optimistic

Real agents violate the independence assumption in two opposite directions, and knowing both is what separates an expert from someone who just quotes pn.

Worked example: what a per-step improvement is worth

Suppose your coding agent's typical task takes about 40 consequential actions, and you measure an unrecoverable-error rate of 2% per action. Predicted end-to-end success is 0.9840 ≈ 0.45. A change that halves the error rate to 1% predicts 0.9940 ≈ 0.67. A 1-point step improvement buys about 22 points end to end. That is why teams chase what look like tiny reliability gains, and why you should not be surprised when an end-to-end eval moves much more than a step-level one.

METR's time horizon: measuring length directly

If long tasks are what matter, one natural question is: how long a task can this agent complete? METR (Model Evaluation and Threat Research, a nonprofit that evaluates frontier models) turned that question into a metric in the paper "Measuring AI Ability to Complete Long Tasks" (Kwa, West et al., March 2025).

The recipe, step by step

1. Build a task suite spanning a huge range of lengths. METR combined three suites: SWAA (very short software actions, seconds long), HCAST (a diverse set of software, ML and cyber tasks of minutes to hours), and RE-Bench (ML research-engineering tasks of about 8 hours). The paper describes these as "RE-Bench, HCAST, and 66 novel shorter tasks".
2. Time human experts. Skilled professionals attempt each task, and the time they take becomes that task's human time. This is the crucial move: task difficulty is measured in a unit everyone understands (minutes of expert labor), not in an abstract score.
3. Run each model many times on each task (roughly 8 runs per model per task, per Epoch AI's summary of the method) and record success or failure.
4. Fit a logistic curve of model success probability against the logarithm of human time. Success is high on short tasks and falls off on long ones.
5. Read off the horizon. The 50% time horizon is the human task length at which the fitted curve crosses 50% success. METR also reports an 80% horizon, which is always shorter.

Formally, with t the human time and σ the logistic (sigmoid) function:

P(success | t) = σ( β0 + β1 · log2(t) )        with β1 < 0
50% horizon    = 2 ^ ( -β0 / β1 )               (where β0 + β1·log2(t) = 0)
q% horizon     = 2 ^ ( (logit(q) - β0) / β1 )   logit(q) = ln(q / (1-q))

Why log time? Because task lengths span five orders of magnitude, from seconds to days, and success falls off roughly evenly per doubling of length, not per extra minute. The slope β1 tells you how sharp the fall-off is: a steep slope means the model is reliable up to a length and then collapses; a shallow slope means it sometimes succeeds even on very long tasks and sometimes fails even on short ones.

The headline result, and what it does and does not mean

In the March 2025 paper, METR reported that Claude 3.7 Sonnet had a 50% time horizon of "around 50 minutes", and that the frontier 50% horizon "has been doubling approximately every seven months since 2019", with a faster pace in 2024 to 2025. METR has since published Time Horizon 1.1 (January 2026), which uses a larger task suite with the same method, and maintains a live page at metr.org/time-horizons (which, when checked on 29 September 2026, said it was last updated 8 May 2026). Current per-model values change often; read them from that page rather than from any course.

Compute a time horizon yourself

This snippet simulates a model whose true 50% horizon is 60 minutes, then recovers it from pass/fail data with scikit-learn. It also computes a bootstrap confidence interval (Module 04) by resampling tasks, because the 8 runs of one task are correlated and resampling runs would make the interval too narrow.

import numpy as np
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(0)

# 1) Synthetic task suite: 120 tasks, each with a human-expert completion time (minutes)
human_minutes = np.exp(rng.uniform(np.log(1), np.log(960), size=120))  # 1 min .. 16 h

# 2) Pretend agent whose "true" 50% horizon is 60 min (we will try to recover it)
TRUE_H50, TRUE_SLOPE = 60.0, 0.7
def p_success(minutes):
    return 1 / (1 + np.exp(TRUE_SLOPE * (np.log2(minutes) - np.log2(TRUE_H50))))

# 3) 8 runs per task (METR used roughly this many), each a Bernoulli draw
RUNS = 8
x = np.repeat(np.log2(human_minutes), RUNS).reshape(-1, 1)  # feature: log2(human time)
y = rng.random(len(x)) < p_success(2 ** x[:, 0])            # 1 = success, 0 = failure

# 4) Fit success ~ sigmoid(b0 + b1 * log2(minutes)); C=np.inf means no regularisation
clf = LogisticRegression(C=np.inf).fit(x, y)
b0, b1 = clf.intercept_[0], clf.coef_[0, 0]

# 5) Solve sigmoid(b0 + b1*t) = q  ->  t = (logit(q) - b0) / b1
def horizon(q, b0, b1):
    return 2 ** ((np.log(q / (1 - q)) - b0) / b1)

print(f"50% horizon: {horizon(0.5, b0, b1):6.1f} min   (true {TRUE_H50})")
print(f"80% horizon: {horizon(0.8, b0, b1):6.1f} min")

# 6) Bootstrap over TASKS (not runs): runs of one task are correlated
tasks = np.arange(len(human_minutes))
h50s = []
for _ in range(500):
    pick = rng.choice(tasks, size=len(tasks), replace=True)
    idx = (pick[:, None] * RUNS + np.arange(RUNS)).ravel()
    m = LogisticRegression(C=np.inf).fit(x[idx], y[idx])
    h50s.append(horizon(0.5, m.intercept_[0], m.coef_[0, 0]))
lo, hi = np.percentile(h50s, [2.5, 97.5])
print(f"95% CI for 50% horizon: {lo:.1f} to {hi:.1f} min")

# Output (numpy 2.4, scikit-learn 1.9):
# 50% horizon:   59.3 min   (true 60.0)
# 80% horizon:   17.8 min
# 95% CI for 50% horizon: 50.2 to 70.6 min

Even with 960 runs, the 95% interval spans roughly 50 to 71 minutes. Real suites have fewer long tasks than this synthetic one, which is exactly why METR's frontier intervals are wide. Try shrinking the task count to 30, or removing all tasks above 4 hours, and watch the interval explode.

Long-task benchmarks and how they score

A family of benchmarks targets tasks that take humans hours or days. What makes them instructive is that each had to invent a scoring scheme, because "did it pass?" is too coarse for an 8-hour task. The details below come from each benchmark's paper.

What these benchmarks teach about long-horizon evals

• Budgets are part of the task. RE-Bench's human-versus-agent result flips depending on whether you allow 2 hours or 32. A long-horizon score without its time, token and compute budget is not interpretable.
• Variance is enormous. Vending-Bench's most striking finding is not the average: it is that the same model can run a profitable business in one run and melt down in the next. Report distributions (or pass^k, Module 04), not just means.
• Graders get harder as tasks get longer. Short tasks have unit tests. Long, open-ended tasks push you toward rubrics and LLM judges (PaperBench) or expensive human experts (GDPval), each of which must itself be validated (Module 13).
• Realism costs reproducibility. GDPval's deliverables are realistic but need expert graders; MLE-bench is reproducible but Kaggle solutions are public, raising contamination questions (Module 18).

Milestones and partial credit

A binary "did it finish?" score has a serious problem on long tasks: it throws away almost all the information. An agent that got 90% of the way through a 6-hour migration and an agent that never compiled the code both score 0. With few long tasks (and each run costing real money), you cannot afford to waste signal. The fix is to define milestones: checkable intermediate states on the path to success, each worth some share of the credit.

Designing good milestones

• Make each milestone checkable by code where possible: "the build passes", "file X contains a call to Y", "the database has row Z". Fall back to a rubric-based LLM judge (Module 13) for fuzzy ones such as "the report explains the method".
• Order and gate them. Gating is what stops reward hacking: without it, an agent can score the "old client removed" milestone by simply deleting code.
• Weight by value, not by effort. Integration tests passing may take less agent time than setting up the build, but it is worth more.
• Keep the binary metric too. Partial credit answers "how much progress?"; binary success answers "could we ship this?". A change that raises partial credit while lowering full success means the agent got better at starting and worse at finishing: very important to know.
• Record "furthest milestone" histograms. If 60% of failures stall at milestone 4, you know exactly where to look (Module 12's error analysis, applied to long tasks).

PaperBench is a large-scale example of the same idea: its rubric trees decompose "replicate this paper" into thousands of weighted leaves, and the score is a weighted average up the tree. RE-Bench's normalized score is a continuous cousin: it gives credit for any improvement over the starting solution.

from dataclasses import dataclass, field
from typing import Callable

@dataclass
class Milestone:
    id: str
    weight: float                      # share of total credit
    check: Callable[[dict], bool]      # programmatic check on the final state / trace
    requires: list[str] = field(default_factory=list)  # gating: only counts if these passed

def score(milestones: list[Milestone], state: dict) -> dict:
    passed: dict[str, bool] = {}
    for m in milestones:               # listed in dependency order
        ok = m.check(state) and all(passed.get(r, False) for r in m.requires)
        passed[m.id] = ok
    total = sum(m.weight for m in milestones)
    earned = sum(m.weight for m in milestones if passed[m.id])
    return {
        "partial_credit": round(earned / total, 3),
        "full_success": all(passed.values()),          # the strict binary metric
        "furthest": max((i for i, m in enumerate(milestones) if passed[m.id]), default=-1),
        "detail": passed,
    }

# Task: "migrate the billing service to the new payments API"
M = [
    Milestone("repo_builds",      0.10, lambda s: s["build_ok"]),
    Milestone("client_swapped",   0.20, lambda s: "new_payments" in s["imports"], ["repo_builds"]),
    Milestone("unit_tests_pass",  0.30, lambda s: s["unit_pass_rate"] == 1.0,     ["client_swapped"]),
    Milestone("integration_pass", 0.30, lambda s: s["integration_ok"],            ["unit_tests_pass"]),
    Milestone("old_client_gone",  0.10, lambda s: "legacy_pay" not in s["imports"], ["integration_pass"]),
]

final_state = {"build_ok": True, "imports": {"new_payments", "legacy_pay"},
               "unit_pass_rate": 1.0, "integration_ok": False}
print(score(M, final_state))
# {'partial_credit': 0.6, 'full_success': False, 'furthest': 2,
#  'detail': {'repo_builds': True, 'client_swapped': True, 'unit_tests_pass': True,
#             'integration_pass': False, 'old_client_gone': False}}

Context, memory and state tracking

A long task produces more text than fits in a model's context window (the maximum number of tokens the model can read at once). Agent harnesses therefore manage context: they compact (summarize older turns), truncate tool outputs, or let the agent write notes to an external memory (a file, a database) and read them back later. Anthropic's research system, for example, saves its plan to external memory when approaching token limits. Each of these is a place where information can silently disappear, so each deserves its own eval.

Three layers of memory evaluation

Long-context evals: beyond the needle

The first row of the table above deserves a closer look, because "the model has a 1-million-token window" is one of the most misread numbers in AI. An advertised context window tells you how much text the model accepts. It does not tell you how much it can use.

The classic test is the needle in a haystack (popularized in 2023 by Greg Kamradt's pressure tests): hide one sentence ("the best thing to do in San Francisco is eat a sandwich in Dolores Park") at some depth inside a long, unrelated document, then ask for it. Sweep the length and the depth, color a grid green where the model found it, and you get the famous all-green heatmaps. The trouble is that this test is easy in ways real work is not:

• Literal match. The question repeats the needle's words, so finding it is closer to text search than to understanding.
• The needle sticks out. A sentence about sandwiches inside essays about startups is conspicuous. Real documents are full of similar-looking facts (last quarter's number next to this quarter's).
• One fact, no reasoning. Real tasks chain facts together or count across the whole input.

RULER (Hsieh et al., NVIDIA, 2024) keeps the synthetic, auto-graded style but adds harder task families, all generated at any length you choose: retrieval variants with multiple keys, multiple values and distracting look-alike needles; multi-hop tracing (variable tracking: "X1 = 12345 ... X2 = X1 ... X3 = X2", then "which variables equal 12345?", which forces following a chain across the whole input); aggregation (find the most common words in a long list, which needs the whole input, not one spot); and question answering with distractor paragraphs. The paper's headline: almost all models tested reached near-perfect scores on the vanilla needle test, yet showed large drops as length grew on the harder tasks, and only about half of the models that claimed 32K or more of context kept satisfactory performance at 32K. RULER calls the longest length at which a model stays above a fixed quality threshold its effective context length, which is the number you actually want.

"Context rot" is the name Chroma's July 2025 technical report gave to the broader pattern: across 18 models, performance degraded as input length grew, even on simple tasks, and not uniformly. The report found degradation was worse when the needle was less semantically similar to the question, that distractors (plausible but wrong near-matches) hurt more as inputs grew, and, counter-intuitively, that the structure of the haystack mattered (models did differently on logically coherent haystacks than on shuffled ones). In a conversational memory test built on LongMemEval, models did markedly better when given only the relevant portion of the history than the full history. The code is public (chroma-core/context-rot), so you can rerun it on your own model.

For an agent, "long context" is rarely one long document. It is a growing transcript of tool outputs, file contents and earlier turns. Build your own long-context eval from that material rather than from essays about startups:

1. Plant facts from your own domain in real transcripts at varying depths (an order number at turn 5, a changed shipping address at turn 40), then ask about them at the end. Use paraphrase, not the same words, between the fact and the question.
2. Add realistic distractors: the old address next to the new one, last week's order next to today's. Score whether the agent picks the current fact.
3. Include aggregation and multi-hop questions: "how many of the tickets so far mention refunds?", "which customer's second order used the coupon from the first email?".
4. Sweep length (for example 8K, 32K, 128K of real context) and plot accuracy against length, as above. Report the effective length for your threshold, not the vendor's window.
5. Compare context strategies on the same items: full context, compaction, retrieval over notes. This is where the state-tracking probes and compaction tests from this section meet (harness side in Module 21).

Cost and wall-clock budgets

Long tasks are expensive: MLE-bench runs take about a day of GPU time each, and Vending-Bench runs exceed 20 million tokens. That has two consequences for evals.

1. The budget is a variable you must fix and report. Agents can often trade money for success: more attempts, more thinking tokens, more subagents. An agent at 60% for $2 per task and one at 65% for $40 per task are not ranked by success alone. Report success at a stated budget, or plot success against cost and look for the Pareto frontier (the set of systems not beaten on both axes at once).
2. You will run fewer samples, so statistics matter more (Module 04). Prefer paired comparisons on the same tasks, report confidence intervals, and use partial credit to squeeze more signal from each run.

Measure at least: tokens in and out (and cached), dollar cost, number of tool calls, wall-clock time, and for multi-agent systems, peak parallelism. Put hard caps in the harness (maximum steps, tokens, minutes) and record why a run ended: finished, gave up, hit the cap, or crashed. "Hit the step cap" is a very different failure from "confidently submitted a wrong answer".

Part B · Multi-agent systems: the architectures

A multi-agent system is any setup where two or more model-driven agents, each with its own instructions and usually its own context window, cooperate on a task by passing messages or artifacts. The four common shapes each suggest where to grade.

A case study: how Anthropic evaluated its multi-agent research system

In June 2025 Anthropic published an engineering post, "How we built our multi-agent research system", describing the architecture behind Claude's Research feature and, unusually, how they evaluated it. It is the best public case study of multi-agent evals in production, so it is worth knowing in detail.

Architecture. An orchestrator-worker design: a lead agent analyses the query, plans, and spawns subagents that search in parallel, each with its own context window; a separate CitationAgent attributes claims to sources at the end.

Headline results (as stated in the post):

• A multi-agent system with Claude Opus 4 as lead and Claude Sonnet 4 subagents outperformed single-agent Claude Opus 4 by 90.2% on Anthropic's internal research eval.
• On BrowseComp, token usage alone explained 80% of the performance variance; token usage, number of tool calls and model choice together explained 95%.
• Agents use about 4× more tokens than chat interactions, and multi-agent systems about 15× more.

How they evaluated it

The post also lists failure modes the team saw in early versions: spawning 50 subagents for simple queries, searching endlessly for sources that do not exist, subagents distracting each other with excessive updates, and subagents duplicating each other's work. They note that "small changes to the lead agent can unpredictably change how subagents behave", and that because agents are stateful over many tool calls, minor failures can cascade. Full production tracing (of decision patterns and interaction structure, not conversation contents) was how they diagnosed these.

What to measure in a multi-agent system

Think in three levels: the whole system, each agent, and the connections between them.

Credit assignment: who deserves the blame?

Credit assignment is the problem of deciding which agent (or which step) caused the outcome. It is hard because agents interact: a worker's wrong fact only matters if the lead failed to cross-check it. Three practical approaches, from cheap to rigorous:

1. Trace inspection with a judge. Give an LLM judge the full trace and ask "which agent's action first made the task unrecoverable?". Cheap and useful for error analysis, but the judge can be wrong, so spot-check it.
2. Counterfactual replay (ablation). Replay the run with one agent's output swapped for a known-good one (from a stronger model or a human). If success jumps, that agent was the bottleneck. This is the most convincing method and also the most expensive.
3. Shapley-style attribution. For a small number of agents, measure the outcome for every subset of "upgraded" agents and average each agent's marginal contribution (the Shapley value from game theory). The cost grows as 2ⁿ, so it is only practical for a handful of agents.

Tracing across agents: span trees

A single-agent transcript is a flat list. A multi-agent run is a tree: the lead's work contains each subagent's work, which contains each tool call. Tracing tools record this as nested spans (a span is a timed unit of work with a parent, attributes and a status). The OpenTelemetry GenAI semantic conventions (still marked "Development", i.e. not yet stable) define span names like invoke_agent {gen_ai.agent.name} and execute_tool, with attributes such as gen_ai.agent.name and token usage counts. Many eval and observability tools (Module 14) ingest these traces, which means you can attach graders to any node of the tree.

The walker below computes per-agent tokens, tool calls and errors from such a tree, reports the ancestor path of every error (so you know which delegation led there), and computes the token multiplier against a single-agent baseline.

from collections import defaultdict

# A multi-agent trace as a tree of spans (the shape OpenTelemetry-style tracers export)
trace = {
  "name": "invoke_agent lead", "agent": "lead", "in_tok": 9_000, "out_tok": 2_500, "error": None,
  "children": [
    {"name": "invoke_agent search_1", "agent": "search_1", "in_tok": 30_000, "out_tok": 3_000, "error": None,
     "children": [
        {"name": "execute_tool web_search", "agent": "search_1", "in_tok": 0, "out_tok": 0, "error": None, "children": []},
        {"name": "execute_tool web_fetch",  "agent": "search_1", "in_tok": 0, "out_tok": 0, "error": "timeout", "children": []}]},
    {"name": "invoke_agent search_2", "agent": "search_2", "in_tok": 42_000, "out_tok": 4_100, "error": None,
     "children": [
        {"name": "execute_tool web_search", "agent": "search_2", "in_tok": 0, "out_tok": 0, "error": None, "children": []}]},
    {"name": "invoke_agent citations", "agent": "citations", "in_tok": 18_000, "out_tok": 1_200, "error": None, "children": []},
  ],
}

def walk(span, depth=0, path=()):
    """Depth-first: yield every span with its depth and ancestor path."""
    yield span, depth, path
    for child in span["children"]:
        yield from walk(child, depth + 1, path + (span["agent"],))

per_agent = defaultdict(lambda: {"tokens": 0, "spans": 0, "errors": 0, "tool_calls": 0})
for span, depth, path in walk(trace):
    a = per_agent[span["agent"]]
    a["tokens"] += span["in_tok"] + span["out_tok"]
    a["spans"] += 1
    a["tool_calls"] += span["name"].startswith("execute_tool")
    if span["error"]:
        a["errors"] += 1
        print(f"error at depth {depth}: {span['name']} ({span['error']}) via {' > '.join(path)}")

total = sum(a["tokens"] for a in per_agent.values())
for name, a in sorted(per_agent.items(), key=lambda kv: -kv[1]["tokens"]):
    print(f"{name:10s} {a['tokens']:>7,} tok ({a['tokens']/total:5.1%})  "
          f"tools={a['tool_calls']}  errors={a['errors']}")

SINGLE_AGENT_BASELINE = 20_000     # tokens the single-agent version used on this task
print(f"token multiplier vs single agent: {total / SINGLE_AGENT_BASELINE:.1f}x")

# Output:
# error at depth 2: execute_tool web_fetch (timeout) via lead > search_1
# search_2    46,100 tok (42.0%)  tools=1  errors=0
# search_1    33,000 tok (30.1%)  tools=2  errors=1
# citations   19,200 tok (17.5%)  tools=0  errors=0
# lead        11,500 tok (10.5%)  tools=0  errors=0
# token multiplier vs single agent: 5.5x

In a real system you would read spans from your tracing backend (exported OpenTelemetry JSON, or a platform's SDK) rather than a hand-written dict, and you would attach graders' scores back onto spans so that dashboards can slice by agent. The logic stays the same: walk the tree, attribute, aggregate.

Why multi-agent systems fail: the MAST taxonomy

When you label failed traces (Module 12), it helps to start from an established taxonomy rather than inventing categories. The most cited one is MAST (Multi-Agent System failure Taxonomy) from "Why Do Multi-Agent LLM Systems Fail?" (Cemri, Pan, Yang et al., UC Berkeley, March 2025). The authors annotated traces from seven open multi-agent frameworks (including ChatDev, MetaGPT, HyperAgent, AppWorld, AG2, Magentic-One and OpenManus), reached high inter-annotator agreement (Cohen's kappa 0.88), and released a dataset of more than 1,600 annotated traces along with an LLM-as-judge pipeline for labeling new ones. They found 14 failure modes in 3 categories.

Three lessons from MAST that change how you build evals:

• Verification is a first-class component. About a quarter of failures are in task verification: the system either never checks its work, checks it badly, or stops early. If your architecture has a "reviewer" or "verifier" agent, evaluate that agent on its own as a classifier (precision and recall on known good and bad outputs), exactly like validating an LLM judge (Module 13).
• Many "multi-agent" failures are single-agent failures in disguise. Step repetition, disobeying the task specification and not knowing when to stop would happen with one agent too. Before blaming coordination, check whether a single agent fails the same way.
• Stopping conditions need tests. "Unaware of stopping conditions" is a big category. Include tasks whose correct behavior is to stop early (the answer is already known, the task is impossible) and measure whether the system does.

Emergent behavior and safety in multi-agent systems

When agents interact, behaviors can appear that no single agent shows on its own. That is what "emergent" means here, and it is why multi-agent systems need evals beyond their parts.

The Cooperative AI Foundation's report "Multi-Agent Risks from Advanced AI" (Hammond et al., February 2025) organizes the risks into three failure modes (miscoordination, conflict, collusion) and seven risk factors (information asymmetries, network effects, selection pressures, destabilising dynamics, commitment problems, emergent agency, and multi-agent security). It is a good checklist for designing adversarial multi-agent evals, and Module 17 returns to dangerous-capability testing more broadly.

Agent interop: A2A, handoffs and agent payments

Everything in Part B so far assumed you own all the agents. Increasingly, your agent will hand work to an agent someone else built and runs, and some agents can now spend money. Two families of open protocols are emerging for this, and both create new things to evaluate. (The protocols are young and change quickly, so treat field names below as illustrative and check the current specs.)

For evaluation, the protocol details matter less than the new failure points they create. A delegation can go to the wrong agent; the user's constraints can be lost in the handoff; the remote agent can ask a question, fail or stall; and a payment can exceed what the user actually authorized. The figure puts numbered checkpoints on each.

Payment evals are a place for hard, deterministic checks, not judges. A sketch of the grader side, run after the agent finishes each scenario against a sandbox rail that records charges instead of moving money:

from dataclasses import dataclass

@dataclass
class Mandate:                      # what the user actually authorized
    max_total: float                # cumulative cap across all purchases
    categories: set[str]            # e.g. {"tickets"}
    confirm_over: float             # single charges above this need explicit user approval

def grade_payments(mandate: Mandate, charges: list[dict], approvals: set[str]) -> dict:
    """charges: recorded by the sandbox rail, e.g.
       {"order_id": "A1", "amount": 180.0, "category": "tickets"}
       approvals: order_ids the (simulated) user explicitly approved."""
    order_ids = [c["order_id"] for c in charges]
    return {
        "under_cap":        sum(c["amount"] for c in charges) <= mandate.max_total,
        "in_category":      all(c["category"] in mandate.categories for c in charges),
        "approved_if_big":  all(c["order_id"] in approvals
                                for c in charges if c["amount"] > mandate.confirm_over),
        "no_double_charge": len(order_ids) == len(set(order_ids)),
    }

# Scenario "price rises after approval": quote was $180, merchant now asks $230.
# The right behavior is to stop and ask the user, so the rail should record no charge.
print(grade_payments(Mandate(200.0, {"tickets"}, 150.0), charges=[], approvals=set()))
# every check True: the agent refused to pay outside the mandate

Run each scenario several times and require every check to pass on every trial (pass^k from Module 06): for money, "usually stays under budget" is a failing grade. Public benchmarks for agent interop and payments are still immature, so most teams build scenario suites like this one around their own flows.

Putting it together: an eval plan for a long-running multi-agent system

Suppose you are building a research assistant like the one in the case study. A minimal but expert-grade eval plan looks like this:

1. Dataset: 20 to 50 real queries to start, stratified by complexity (simple fact, comparison, deep research), each with a rubric and, where possible, known key facts and sources.
2. Baseline: a single-agent version with the same model and the same total token budget.
3. Outcome graders: a rubric LLM judge (accuracy, citation accuracy, completeness, source quality, tool efficiency), validated against human labels on a subset. Milestones for long queries (sources found, claims extracted, draft written, citations verified).
4. Process graders on the trace: subagent count vs complexity, duplicated queries across workers, tool error and recovery rate, MAST labels on failures.
5. Budgets: tokens, dollars, wall-clock, subagent count; report the multiplier vs baseline and plot quality against cost.
6. Robustness: 3 or more runs per query (report pass^k or spread), fault injection on one worker, prompt injection via a planted web page.
7. Human review: a weekly read of a sample of traces, looking for things the rubric does not capture (the content-farm lesson).

Check yourself

Sources and further reading

• Kwa, West et al. (METR), Measuring AI Ability to Complete Long Tasks (March 2025).
• METR, Time horizons (live page), Time Horizon 1.1 (January 2026), Clarifying limitations of time horizon, Impact of modeling assumptions.
• Epoch AI, METR time horizons mirror.
• Wijk et al. (METR), RE-Bench (November 2024).
• Chan et al. (OpenAI), MLE-bench (October 2024).
• Starace et al. (OpenAI), PaperBench (April 2025).
• OpenAI, GDPval (September 2025).
• Backlund and Petersson (Andon Labs), Vending-Bench (February 2025); Andon Labs Vending-Bench page.
• Hsieh et al., RULER: What's the Real Context Size of Your Long-Context Language Models? (2024); Liu et al., Lost in the Middle (2023).
• Chroma, Context Rot: How Increasing Input Tokens Impacts LLM Performance (technical report, July 2025) and code; Greg Kamradt, Needle in a haystack pressure tests (2023); NVIDIA RULER repository.
• A2A project, A2A protocol specification and docs and GitHub; Linux Foundation, A2A protocol surpasses 150 organizations.
• Google Cloud, Announcing Agent Payments Protocol (AP2) (September 2025); AP2 documentation.
• OpenAI, Buy it in ChatGPT: Instant Checkout and the Agentic Commerce Protocol; Stripe, Developing an open standard for agentic commerce; ACP specification on GitHub.
• Coinbase, Introducing x402 and x402 repository.
• Anthropic, How we built our multi-agent research system (June 2025).
• Cemri, Pan, Yang et al., Why Do Multi-Agent LLM Systems Fail? (MAST, March 2025).
• Du et al., Improving Factuality and Reasoning in Language Models through Multiagent Debate (2023).
• Motwani et al., Secret Collusion among AI Agents: Multi-Agent Deception via Steganography (2024).
• Hammond et al. (Cooperative AI Foundation), Multi-Agent Risks from Advanced AI (February 2025).
• OpenTelemetry, GenAI semantic conventions (agent and tool spans; status: Development).