Why agents are different

A chatbot answers. An agent acts: it takes many steps, calls tools, changes the state of the world, and decides for itself when it is done. That single shift, from producing text to producing consequences, changes what you grade, where you run the test, how you count success, and what can go wrong while you are measuring.

What an agent actually is

In Modules 01 to 04 the thing under test mostly looked like a function: text in, text out, grade the text. An agent breaks that shape. Here is a working definition you can defend in any interview:

Five parts, each of which will become something you have to control or measure in an eval:

• Model: the LLM that reads the current context and decides the next action. It is the "brain", but it is only one component.
• Tools: functions the model can call, described to it by a name, a description and a parameter schema. Examples: search_flights, bash, click(x, y), send_email. A tool call is just structured text (usually JSON) that the surrounding program executes.
• Loop: the program around the model (often called the harness or scaffold) that sends context to the model, executes the tool call it asks for, appends the result, and repeats until the model says it is finished or a limit is hit.
• Environment: the world the tools act on. A database, a code repository, a web browser, a whole virtual machine, a phone, a live phone call. Crucially, the environment has state, and actions change it.
• Memory: whatever carries over between steps. At minimum, the growing conversation context. Often also scratch files, a notes tool, a vector store, or summaries written when the context window fills up.

Workflows vs agents

Anthropic's widely cited essay Building effective agents (Dec 2024) draws a line that is useful for evals too. Its building block is the augmented LLM: "an LLM enhanced with augmentations such as retrieval, tools, and memory." From there it separates two kinds of systems:

• Workflows: "systems where LLMs and tools are orchestrated through predefined code paths." The patterns it names are prompt chaining, routing, parallelization, orchestrator-workers and evaluator-optimizer.
• Agents: "systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks."

Why does this matter for evals? Because it tells you what you can check. For a workflow you know the steps in advance, so you can evaluate each LLM call in isolation, like a unit test per stage, and compose the results. For an agent the number and order of steps is open-ended. The essay recommends agents for "open-ended problems where it's difficult or impossible to predict the required number of steps", which is exactly the property that makes step-by-step testing brittle. It also stresses that agents must "gain 'ground truth' from the environment at each step (such as tool call results or code execution)" and recommends "extensive testing in sandboxed environments". Those two phrases are, in effect, the design brief for this module.

The agent loop as a state machine

A useful mental model is to draw the loop as a state machine: a set of states and the transitions between them. It forces you to answer a question that beginners forget: in how many ways can a run end? Each ending is something your eval must detect and score.

Three things fall out of this picture:

1. The trajectory is the object of study. A trajectory (also called a transcript or trace) is the complete ordered record of a trial: every message, every tool call with its arguments, every tool result, every intermediate thought the harness exposes, and timestamps and token counts. Anthropic's engineering post Demystifying evals for AI agents (Jan 2026) defines it as "the complete record of a trial, including outputs, tool calls, reasoning, intermediate results, and any other interactions."
2. "Finish" is a decision, not a fact. The model deciding it is done does not mean the task is done. Premature "I've completed the booking!" messages, with no booking in the database, are one of the most common agent failures. This is why you grade the environment, not the claim.
3. Non-success exits are data. Tag each trial with its exit reason (finished, max steps, timeout, token limit, tool crash, harness crash). A spike in "max steps" after a prompt change is a loop bug; a spike in "harness crash" is an infrastructure bug that should not count against the model at all.

Anatomy of an agent eval

Module 02 dissected a plain eval into dataset, system, grader and metric. An agent eval has the same skeleton, but several parts grow new organs. Using the vocabulary from Anthropic's post (which matches Inspect, tau-bench and most frameworks):

• Task: "a single test with defined inputs and success criteria." For agents the inputs include not only an instruction but an initial environment state (called s0 here) and often a hidden goal for a simulated user.
• Trial: one attempt at a task. "Because model outputs vary between runs, we run multiple trials to produce more consistent results."
• Environment: the sandbox the agent acts in, with its tools, data and rules, created fresh for every trial.
• Agent harness (scaffold): the loop and prompt around the model. It is part of the system under test. The separate evaluation harness is the infrastructure that runs trials, collects trajectories and calls graders.
• Simulated user (when needed): a second model that plays the human, for tasks where the agent must ask questions or confirm actions.
• Outcome: "the final state in the environment at the end of the trial."
• Graders: "logic that scores some aspect of the agent's performance. A task can have multiple graders, each containing multiple assertions."

Here is what a single task specification might look like as data. The exact format differs by framework (Inspect Sample, tau-bench task JSON, ADK .evalset.json), but the fields are almost always these:

# One task in a travel-booking agent eval, expressed as plain data.
task = {
    "id": "change-flight-017",
    # What the agent is told (or what the simulated user will say first).
    "instruction": "Customer wants to move booking R7K2 to Friday.",
    # Environment seed: the exact database the sandbox starts from (s0).
    "initial_state": "fixtures/db_change_flight_017.json",
    # Only the simulated user sees this. The agent must discover it by asking.
    "user_scenario": {
        "persona": "busy, terse, answers only what is asked",
        "hidden_goal": "Move R7K2 (JFK-CDG) to Fri 9 Oct, keep economy. "
                       "Accept a fare difference only if it is under $100.",
    },
    # Success criteria, checked by graders after the run.
    "expected_end_state": {
        "reservations.R7K2.flight": "AF125",
        "reservations.R7K2.cabin": "economy",
        "payments.count_new": 1,
        "payments.new[0].amount": 80,
    },
    "must_communicate": ["new flight number", "fare difference"],
    "forbidden_actions": ["cancel_reservation", "refund_payment"],
    # Budgets: runs that exceed these end as failures, and are tagged as such.
    "limits": {"max_steps": 30, "max_tokens": 200_000, "timeout_s": 600},
}

Three layers of grading

For a plain LLM output there is one thing to grade: the output. An agent leaves three artifacts behind, and each supports a different kind of grader.

Layer 1: outcome (end-state) grading

After the run, inspect the environment. Did the reservation change? Do the hidden unit tests pass? Is the file on disk, with the right content? Is there exactly one new payment, not two? This is what tau-bench does (it compares the final database with an annotated goal database), what SWE-bench does (it runs FAIL_TO_PASS and PASS_TO_PASS tests), what OSWorld does (134 custom evaluation functions that inspect file and app state) and what AppWorld does (state-based unit tests over the database diff, including checks for "collateral damage", meaning unintended changes).

Outcome grading is robust for a deep reason: it is indifferent to the route. It asks the question the user actually cares about. It is also usually cheap and deterministic, because it is code that reads a database or runs tests (see Module 03).

Its weaknesses: some tasks have no checkable end state (research, advice, explanations); a correct end state can be reached by an unacceptable path (it booked the right flight but also emailed the customer's passport to a stranger); and end-state checks can be too narrow. A check that looks only at reservations.R7K2 misses that the agent also canceled R7K3. Good outcome graders check the whole diff between s0 and the end state: what should have changed did, and nothing else changed.

Layer 2: process (trajectory) grading

Here you read the trajectory itself. Typical checks:

• Tool choice: did it call the right tools? Did it call a tool that was not needed?
• Argument correctness: were the parameters valid and right (the correct booking id, a date in the right format, no invented fields)? BFCL, the Berkeley Function Calling Leaderboard, grades single calls this way using AST matching (comparing the parsed call's structure against allowed values).
• Order: did a precondition come before the action (look up, then modify; ask for confirmation, then charge)?
• Redundancy and efficiency: repeated identical calls, loops, wasted steps. These rarely change correctness but they change cost and latency.
• Policy violations: actions the domain policy forbids (refunding without authorization, touching another customer's data, running rm -rf outside the working directory).

Trajectory metrics have been standardized in some tools. Google's Vertex AI Gen AI evaluation service offers trajectory_exact_match, trajectory_in_order_match (all reference calls present in the same order, extra calls allowed), trajectory_any_order_match (all reference calls present in any order, extras allowed), trajectory_precision, trajectory_recall and trajectory_single_tool_use (was one named tool used at all). Google's Agent Development Kit (ADK) has a tool_trajectory_avg_score criterion with match_type set to EXACT, IN_ORDER or ANY_ORDER. You will write your own versions of these in the worked example below.

The rule of thumb: grade outcomes for correctness, grade process for constraints. Process checks should mostly be negative ("never call refund without a prior confirmation", "no more than 3 identical calls") or necessary conditions ("must call get_reservation before modifying it"), not a complete script. Reserve exact or in-order matching for places where the order truly is the requirement, such as regulated flows, or for fast unit tests of a narrow sub-agent where only one reasonable path exists.

Layer 3: response grading

Most agents end with a message to the user. It can be wrong even when the world is right ("your flight is on Thursday" when the database says Friday), or it can omit something the user needed to know (the fare difference). tau-bench adds exactly this: besides the database check, it checks that required information was communicated to the user. Response grading uses the techniques from Modules 03 and 13: string checks for required facts, rubrics, or an LLM judge. The most valuable response check is consistency with the end state: every claim in the final message should be true of the environment, which catches the classic "claims success, did nothing" failure.

Evaluating planning

Many agent tasks are really planning problems in disguise. "Book me a three-day trip to Denver under $1,200, the hotel must take my dog, and no flights before 8 a.m." has a goal, several constraints, and many possible sequences of actions, most of which are wrong. Planning is the agent's ability to choose and order the steps that reach a goal while respecting every constraint, and to fix that sequence when the world surprises it. It is one of the building blocks of every capable agent, and it deserves its own measurements.

What "planning" means for an LLM agent

LLM agents plan in three recognizable styles, and most real agents mix them:

Why final-outcome scoring hides planning failures

Earlier in this module the rule was "grade outcomes for correctness". That rule still holds, but a pass/fail outcome score cannot tell you why an agent passed or failed, and for planning the "why" matters:

• Lucky success. The plan ignored the budget, but the first flight it found happened to be cheap. The outcome passes; on a harder task the same plan fails.
• Unlucky failure. The plan was perfect, but a tool timed out. The outcome fails, and the fix belongs to the environment, not the planner.
• Expensive success. The agent reached the goal after 40 steps, including three abandoned plans. The outcome passes; your cost bill does not.
• Unchecked constraints. The end-state check verifies "a flight and a hotel are booked" but nobody wrote a check for "no flights before 8 a.m.". The plan broke a constraint that the outcome grader never looked at.

So planning evaluation adds a second question next to "did it work?": "was the plan valid, good, followed, and repaired well?" The picture below shows where each measurement attaches to the loop.

What to measure

Six measurements cover almost everything people mean by "good planning". You rarely need all six for one agent; pick the ones your failures point to (Module 12).

Two definitions you will meet in papers. A plan is valid (or sound) if executing it from the start state reaches the goal without breaking any rule. It is optimal if no valid plan has lower cost. Validity is a yes/no property you can usually check with code. Optimality needs you to know the best possible cost, which you often do not, so in practice people report the cost ratio: the agent's plan cost divided by the best known cost (1.0 is best).

How to grade a plan

From most to least trustworthy:

1. Programmatic constraint checkers (best). Write each hard constraint as a small function of the plan: total_cost(plan) <= budget, hotel.pets_allowed, every flight departs at or after 08:00. Each one returns pass or fail, deterministically, for any plan the agent could produce, including ones you never imagined. This is exactly the "grade constraints, not routes" rule from the process-grading section above, applied to plans. The catch: you must extract a structured plan (a JSON itinerary, a list of tool calls) from the agent, so ask for structured output or parse the trajectory.
2. Formal validators, for classical planning. When the task is written in PDDL (Planning Domain Definition Language, the standard input format for classical AI planners: a domain file with actions, their preconditions and effects, and a problem file with a start state and goal), a plan validator such as VAL simulates the plan step by step and reports whether every precondition held and the goal was reached. PlanBench grades LLM plans this way. It is a perfect checker, but only for tasks you can formalize.
3. Reference-plan comparison, with care. Comparing against a gold plan is tempting because it is easy: step overlap, in-order match, edit distance. The trouble is that most real tasks have many valid plans. An agent that takes a different flight, or books the hotel first, is not wrong. Use a reference plan for quality (is the agent's cost close to the reference cost?) and as a hint for a judge, not as the definition of correct. This is the over-constraint trap from earlier in this module, wearing a planning hat.
4. LLM judge with a rubric, for soft qualities. Some things resist code: "Is the decomposition sensible?", "Was the deviation from the plan justified by what the agent saw?", "Does the itinerary respect commonsense (no three museums scheduled at the same hour)?". A judge with a specific rubric can score these (see Module 13), but validate the judge against human labels first, and never let it overrule a failed hard-constraint check.

Worked example: three plans, three hard constraints

The task: "Plan a trip from Seattle to Denver for 2 people, 2 nights. Hard constraints: C1 total cost at most $1,200; C2 the hotel allows pets; C3 no flight departs before 08:00." Three agent runs produce three plans (illustrative numbers):

(Total = flight fares × 2 travelers + hotel × 2 nights. Plan A: (260 + 240) × 2 + 75 × 2 = 1,150.)

There are two honest ways to summarize this table, and they give very different numbers:

• Micro pass rate (per constraint): count every individual check. 9 checks, 6 passed, so 6 / 9 = 66.7%.
• Macro pass rate (per plan): a plan counts only if it passes all its constraints. 1 of 3 plans, so 1 / 3 = 33.3%.

Micro tells you how close the agent is getting: it gives credit for fixing one constraint at a time, which is useful while you iterate on a prompt. Macro tells you what the user experiences: a plan that breaks any hard constraint is a plan they cannot use. Report both, and gate on macro. TravelPlanner reports exactly this pair for each constraint family, and its headline "final pass rate" is stricter still: a plan must pass every hard and every commonsense constraint.

Code: score plans and plan adherence

The snippet below uses only the Python standard library. Part 1 turns each hard constraint into a function and computes micro and macro pass rates for the three plans above. Part 2 takes Plan A as the agent's written plan, compares it with what the agent actually did, and labels every off-plan step as a failed plan step, a justified replan (it follows a failed call), a wasted repeat, or an unjustified deviation. Finally it re-checks the constraints on what was actually booked, because a replan can quietly break a constraint the original plan satisfied.

# Plan evaluation with only the standard library.
# Part 1 scores plans against hard constraints (micro and macro pass rate).
# Part 2 scores plan adherence against an execution trace.

TRAVELERS, NIGHTS, BUDGET = 2, 2, 1200

def total_cost(plan):
    fares = sum(f["price"] for f in plan["flights"]) * TRAVELERS
    return fares + plan["hotel"]["per_night"] * NIGHTS

# Each hard constraint is a small, deterministic function of the plan.
CONSTRAINTS = {
    "budget":      lambda p: total_cost(p) <= BUDGET,
    "pets_ok":     lambda p: p["hotel"]["pets_allowed"],
    # zero-padded "HH:MM" strings compare correctly as text
    "no_early":    lambda p: all(f["depart"] >= "08:00" for f in p["flights"]),
}

def check(plan):
    return {name: bool(rule(plan)) for name, rule in CONSTRAINTS.items()}

def pass_rates(plans):
    results = {name: check(p) for name, p in plans.items()}
    cells = [ok for r in results.values() for ok in r.values()]
    micro = sum(cells) / len(cells)                                # per constraint
    macro = sum(all(r.values()) for r in results.values()) / len(results)  # per plan
    return results, micro, macro

plans = {
    "A": {"flights": [{"id": "UA318", "depart": "09:10", "price": 260},
                      {"id": "UA415", "depart": "14:30", "price": 240}],
          "hotel": {"name": "Pine Lodge", "per_night": 75, "pets_allowed": True}},
    "B": {"flights": [{"id": "DL102", "depart": "10:05", "price": 290},
                      {"id": "DL233", "depart": "16:00", "price": 270}],
          "hotel": {"name": "Aspen Inn", "per_night": 95, "pets_allowed": True}},
    "C": {"flights": [{"id": "AS900", "depart": "06:45", "price": 210},
                      {"id": "AS911", "depart": "12:20", "price": 200}],
          "hotel": {"name": "Metro Stay", "per_night": 80, "pets_allowed": False}},
}

results, micro, macro = pass_rates(plans)
for name, r in results.items():
    marks = "  ".join(f"{c}={'pass' if ok else 'FAIL'}" for c, ok in r.items())
    cost = f"${total_cost(plans[name]):,}"
    print(f"Plan {name}: {cost:>6}  {marks}")
print(f"micro pass rate: {micro:.1%}   macro pass rate: {macro:.1%}")

# ---- Part 2: did execution follow Plan A, and were deviations justified? ----
plan_steps = [("search_flights", "SEA-DEN"), ("book_flight", "UA318"),
              ("search_hotels", "Denver"), ("book_hotel", "Pine Lodge"),
              ("send_itinerary", "user")]

trace = [("search_flights", "SEA-DEN", "ok"),
         ("book_flight", "UA318", "error: sold out"),
         ("book_flight", "UA322", "ok"),          # agent replans to a later flight
         ("search_hotels", "Denver", "ok"),
         ("search_hotels", "Denver", "ok"),       # identical repeat
         ("book_hotel", "Pine Lodge", "ok"),
         ("send_itinerary", "user", "ok")]

def adherence(plan_steps, trace):
    """Greedy in-order match: a plan step counts only if it ran successfully."""
    used, i = set(), 0
    for step in plan_steps:
        for j in range(i, len(trace)):
            if trace[j][:2] == step and trace[j][2] == "ok":
                used.add(j); i = j + 1
                break
    labels = []
    for j, (tool, arg, status) in enumerate(trace):
        if j in used:
            continue
        prev = trace[j - 1] if j else None
        if (tool, arg) in plan_steps and status != "ok":
            labels.append((tool, arg, "failed plan step"))
        elif prev and prev[:2] == (tool, arg):
            labels.append((tool, arg, "wasted: exact repeat"))
        elif prev and prev[2] != "ok":
            labels.append((tool, arg, "justified: replan after failure"))
        else:
            labels.append((tool, arg, "UNJUSTIFIED deviation"))
    return len(used) / len(plan_steps), labels

score, labels = adherence(plan_steps, trace)
print(f"\nadherence to plan A: {score:.0%} of planned steps executed in order")
for tool, arg, why in labels:
    print(f"  off-plan: {tool}({arg}) -> {why}")

# After a replan, re-check the constraints on what was actually booked.
executed = {"flights": [{"id": "UA322", "depart": "11:40", "price": 275},
                        {"id": "UA415", "depart": "14:30", "price": 240}],
            "hotel": plans["A"]["hotel"]}
print(f"executed itinerary: ${total_cost(executed):,}, checks {check(executed)}")

Running it prints:

Plan A: $1,150  budget=pass  pets_ok=pass  no_early=pass
Plan B: $1,310  budget=FAIL  pets_ok=pass  no_early=pass
Plan C:   $980  budget=pass  pets_ok=FAIL  no_early=FAIL
micro pass rate: 66.7%   macro pass rate: 33.3%

adherence to plan A: 80% of planned steps executed in order
  off-plan: book_flight(UA318) -> failed plan step
  off-plan: book_flight(UA322) -> justified: replan after failure
  off-plan: search_hotels(Denver) -> wasted: exact repeat
executed itinerary: $1,180, checks {'budget': True, 'pets_ok': True, 'no_early': True}

Read the output as a planning report card. Plan quality: only one plan of three is usable (macro 33%) even though two thirds of individual checks pass. Adherence: the agent executed 4 of 5 planned steps as written; the one it did not was blocked by a sold-out flight, and its replacement flight was a justified replan. Efficiency: one wasted, repeated hotel search. Replanning quality: the revised itinerary still passes all three constraints, at $1,180, just under budget. The outcome grader alone would have said only "pass".

The "justified" rule here is deliberately crude: any off-plan call right after a failed call counts as justified. Real deviations are justified by what the agent observed (a price change, a new user request), so production graders usually hand the plan, the observation and the deviation to an LLM judge with a rubric question such as "Given this observation, was changing the plan necessary, and does the new step serve the original goal?"

Benchmarks that test planning

• PlanBench (Valmeekam et al., 2022, arXiv 2206.10498). Built from classical planning domains such as Blocksworld and Logistics, written in PDDL and translated into natural language. It tests several abilities separately: generating a plan, generating a cost-optimal plan, verifying a plan, predicting the result of executing a plan, reusing a plan, and replanning after an unexpected change. Plans are checked with the VAL validator, so grading is exact. It also includes obfuscated "Mystery" versions of the domains, where action and object names are replaced with meaningless words, to test whether a model plans or recalls familiar text.
• TravelPlanner (Xie et al., 2024, arXiv 2402.01622). A sandbox with tools over roughly four million records (flights, restaurants, attractions, accommodations, distances) and 1,225 travel queries with reference plans. Plans are graded by code against two families of constraints: hard constraints stated by the user (budget, room type and rules, cuisine, transportation) and commonsense constraints that any sensible plan must meet (complete information, no invented entities outside the sandbox, a reasonable city route, no repeated restaurants or attractions, non-conflicting transportation, minimum-night stays). It reports delivery rate, micro and macro pass rates for each family, and a final pass rate that requires everything. The paper reports that a GPT-4 agent reached a final pass rate of only 0.6%, a clear demonstration that decent micro scores can hide near-zero usable plans.
• Natural Plan (Zheng et al., Google DeepMind, 2024, arXiv 2406.04520). Three tasks in natural language: trip planning, meeting planning and calendar scheduling. It supplies the outputs of tools such as Google Flights, Maps and Calendar directly in the prompt, which isolates planning from tool use: a failure is a planning failure, not a search failure. The paper reports that the strongest models it tested solved only about a third of trip-planning problems, and that accuracy falls as problems get more complex (more cities, more people).
• Implicit planning tests. Many agent benchmarks never grade a plan directly but cannot be passed without one. tau-bench requires ordering lookups, policy checks, confirmation and writes correctly across a conversation (Module 06). Long-horizon suites stretch plans over dozens or hundreds of steps, where decomposition, state tracking and replanning dominate the failure analysis (Module 11). And whether a planning component in the harness (a to-do file, an explicit plan step) actually helps is a harness question you answer with an ablation (Module 21).

Environment design

For a chatbot eval, "the environment" is a prompt. For an agent eval, it is a small world that you have to build, freeze and reset. Most of the engineering effort in serious agent evals goes here, and most of the silent bugs live here too. Benchmark maintainers keep publishing "Verified" editions (OSWorld-Verified, WebArena-Verified, tau2's "75+ task fixes") largely to repair broken environments and graders.

The seven design decisions

Multi-turn tasks and simulated users

Many real agents talk to a person while they work: they must ask for a missing booking id, clarify an ambiguous request, get explicit confirmation before charging a card. You cannot put a human in every eval trial, so the standard move is a user simulator: another LLM prompted to play the customer.

The canonical example is tau-bench (τ-bench; Yao et al., Sierra, 2024). Each task gives the simulated user a hidden instruction (who they are, what they want, what they know). The agent has API tools and a domain policy document (for example, airline rules on changes and refunds). At the end, the database is compared with an annotated goal state. The follow-up τ²-bench (2025) adds a "dual-control" telecom domain in which the user also acts on a shared environment (their phone) through their own tools, plus ablations that separate reasoning errors from communication errors.

Pitfalls, and how practitioners handle them

• The simulator adds its own noise. The tau-bench notes list simulator hallucination and early termination as known issues. Because the simulator is stochastic too, variance in scores is agent variance plus simulator variance. Run more trials, and keep the simulator model and prompt pinned across comparisons.
• Errors in ground truth. Several labs reported fixing airline tasks in tau-bench, and the tau2 repository lists "75+ task fixes". A wrong expected end state makes a correct agent fail. Audit the tasks your best agents fail most consistently: some of them are broken tasks.
• Who is being tested? If the simulator is a strong model, it can quietly compensate for a weak agent (rephrasing, offering the id unprompted). If it is weak, it can sink a good agent. τ²-bench's no-user and oracle-plan modes are designed to separate these effects.
• Validate the simulator. Treat it like any other LLM grader (Module 13): sample transcripts, have a human label whether the simulator stayed in character and followed its hidden goal. ADK even ships a criterion for this, per_turn_user_simulator_quality_v1.
• Diversity. Vary personas (terse, chatty, confused, non-native speaker, adversarial) and make sure some hidden goals are impossible under policy, so the correct behavior is to refuse politely.

Non-determinism compounds over steps

This is the single most important quantitative intuition about agents. Suppose each step of a task has a 95% chance of being done correctly and any wrong step ruins the task. Assuming steps fail independently, the chance that a 20-step task succeeds is:

P(success) = 0.95²⁰ ≈ 0.358

A model that is "95% accurate" per action finishes only about a third of 20-step tasks. At 99% per step you get 0.99²⁰ ≈ 0.818. At 90% you get 0.90²⁰ ≈ 0.122.

The independence assumption is a simplification, and it cuts both ways. Real agents can recover: they notice a failed tool call and retry, which makes long tasks more survivable than pⁿ predicts. They can also cascade: one wrong early assumption (the wrong booking id) poisons every later step, and errors are correlated. Both effects are why agent evals must run full end-to-end trajectories rather than multiplying per-step accuracies measured in isolation. The lesson that survives is qualitative and important: long-horizon success is extremely sensitive to per-step reliability. METR's time-horizon work (Module 11) is built on this relationship between task length and success.

Reliability across repeated trials: pass@k vs pass^k

Non-determinism also shows up across trials of the same task. Module 04 covered the two metrics in depth; the short version:

• pass@k: probability that at least one of k trials succeeds. Right when something downstream picks the winner (a human reviews candidates, tests select a patch). It rises with k.
• pass^k (from tau-bench): probability that all k trials succeed. Right for an autonomous agent serving thousands of customers, where every run must work. It falls with k. The tau-bench abstract reports that even state-of-the-art function-calling agents of the time (like gpt-4o) succeeded on fewer than half the tasks and scored below 25% pass^8 in the retail domain.

If your agent succeeds 70% of the time on a task, independently each time, then pass@3 = 1 − 0.3³ ≈ 0.973 while pass^3 = 0.7³ ≈ 0.343. Same agent, same data, a factor of almost three between the two numbers. Always say which one you report.

Budgets: cost, latency, steps, tokens

An agent that would eventually solve everything given unlimited time is not a useful agent. Every agent eval needs explicit limits, and those limits are part of the definition of the benchmark:

• Step budget: maximum number of loop iterations. OSWorld results are commonly reported at 15, 50 or 100 steps, and scores differ a lot between them. Comparing a 15-step result with a 100-step result is comparing different benchmarks.
• Wall-clock timeout: per tool call and per trial. Protects you from hung processes and infinite loops.
• Token limits: both the context window (what happens when the trajectory no longer fits? truncation, summarization and memory strategies are part of the scaffold) and a total token budget.
• Money: report cost per task alongside success. The paper AI Agents That Matter (Kapoor et al., 2024) argues that agent benchmarks should be cost-controlled, showing that simple baselines can match elaborate agents at a fraction of the cost, and that accuracy-only leaderboards reward spending more compute.
• Latency: for interactive and voice agents (Module 09), time to first response and total time are quality metrics in their own right.

Evaluating the scaffold vs the model

Here is a fact that surprises many beginners: an agent benchmark score belongs to a model and a scaffold and a budget, never to a model alone. Change the tools, the prompts, the retry logic or the context management, and the same model gets a different score.

Published evidence:

• SWE-agent (Yang et al., 2024) introduced the idea of an agent-computer interface (ACI): commands and feedback designed for the model the way a good UI is designed for people. Giving GPT-4 Turbo purpose-built commands for viewing, searching and editing files reached 12.5% pass@1 on full SWE-bench, which the paper says "far exceed[s] the previous state-of-the-art achieved with non-interactive LMs."
• Anthropic's SWE-bench write-up for the upgraded Claude 3.5 Sonnet (49% on SWE-bench Verified, with a scaffold of just a bash tool and an edit tool) states directly that "the performance of an agent on SWE-bench can vary significantly based on this scaffolding, even when using the same underlying AI model."
• mini-SWE-agent reached 65% on SWE-bench Verified in about 100 lines of Python (Jul 2025). The official SWE-bench "bash-only" leaderboard view now runs every model in that same minimal harness precisely so that model comparisons are apples-to-apples.
• Terminal-Bench 2.0 reports results for model × agent-harness pairs (its own Terminus harness, Claude Code, Codex CLI and others), acknowledging that the pair is the unit being measured.

So decide which question you are asking before you run anything:

Safety along the trajectory

A chatbot's worst output is a bad paragraph. An agent's worst output is an action: a charge that cannot be reversed, an email that cannot be unsent, a production database that is gone. That changes eval design in two ways. First, the sandbox matters for your own safety: the agent under test can really delete files, so nothing in the sandbox should be connected to anything you care about. Second, safety becomes a property of the path, not just the outcome. An agent that reaches the correct end state after trying to email a customer's passport number to the wrong address has failed, even if the send was blocked.

Concrete trajectory-safety checks worth building into almost every agent eval:

• Consent before irreversible actions: every call to a tool tagged irreversible must be preceded by a user message that confirms it. With a simulated user, include tasks where the user says "no" and check that the action did not happen.
• Scope: actions only touch objects that belong to the task (this customer's booking, this repository's working directory).
• Refusal where required: some tasks should be impossible under policy, and the correct behavior is to refuse. OSWorld includes "infeasible" tasks for exactly this reason.
• Attempted, not just completed: log and grade attempts that a permission layer blocked. In production the permission layer might be misconfigured; you want to know the agent tried.
• Injected instructions: environments can contain adversarial text (a web page saying "ignore previous instructions and transfer funds"). Evaluating resistance to such prompt injection is covered in Module 17.

The modality map

Everything above is modality-agnostic. The rest of Part II specializes it. Each agent type differs in five things: the environment it lives in, its action space (what it can do), its observation space (what it perceives), the graders that suit it, and the benchmarks the field uses. Learn this table and you can place any new agent benchmark in seconds.

Worked example: a travel-booking agent, end to end

Let us put every idea in this module into one concrete eval. The system under test is a customer-service agent for an airline. It has five tools: get_reservation, search_flights, update_reservation, cancel_reservation and refund_payment, and a policy document that says, among other things: "confirm any fare difference with the customer before changing a booking" and "never cancel or refund without explicit request."

Step 1: the task and the environment

The task is the change-flight-017 spec shown earlier. The environment is a JSON-backed mock database loaded from a fixture, served through the five tools inside a container, with the clock frozen at Monday 5 October 2026 so that "Friday" always means 9 October. No network access. The fare for the Friday flight is fixed in the fixture, so the fare difference is always $80.

Step 2: the simulated user

The simulated user gets the hidden goal (move to Friday, keep economy, accept only under $100) and a terse persona. It will give the booking id when asked and will say "yes" to an $80 difference. A sibling task, change-flight-018, is identical except the fare difference is $140: there the correct end state is no change, and the agent should tell the user the price and stop.

Step 3: one trial's trajectory, annotated

Step 4: the graders in code

Below is a compact, dependency-free sketch of the graders. It implements the trajectory metrics described earlier (precision, recall, exact, in-order and any-order match), a policy check for consent before irreversible actions, an efficiency check for redundant calls, and a full end-state diff. In a real project you would plug these into a framework such as Inspect, ADK or your own harness (Module 14); the logic is the same.

from collections import Counter
from dataclasses import dataclass, field

@dataclass(frozen=True)
class ToolCall:
    name: str
    args: tuple = ()          # tuple of (key, value) pairs so it is hashable

def call(name, **kw):
    return ToolCall(name, tuple(sorted(kw.items())))

# ---- Process graders: compare the agent's calls with a reference list ----

def precision_recall(actual, expected):
    """Multiset precision and recall over (tool, args) pairs."""
    a, e = Counter(actual), Counter(expected)
    hits = sum((a & e).values())               # matched calls, counting duplicates
    precision = hits / max(len(actual), 1)       # how many calls were useful
    recall = hits / max(len(expected), 1)        # how many required calls happened
    return precision, recall

def exact_match(actual, expected):
    return actual == expected

def in_order_match(actual, expected):
    """All expected calls appear in order; extra calls in between are allowed."""
    it = iter(actual)
    return all(any(x == y for x in it) for y in expected)

def any_order_match(actual, expected):
    """All expected calls appear (with multiplicity), in any order."""
    return not (Counter(expected) - Counter(actual))

def redundant_calls(actual):
    """Identical calls repeated back to back: a cheap efficiency signal."""
    return sum(1 for prev, cur in zip(actual, actual[1:]) if prev == cur)

IRREVERSIBLE = {"update_reservation", "cancel_reservation", "refund_payment"}

def consent_before_irreversible(events):
    """events: list of ("user", text) | ("tool", ToolCall). Policy check."""
    consented = False
    for kind, payload in events:
        if kind == "user" and payload.strip().lower().startswith(("yes", "ok", "confirm")):
            consented = True            # crude; use an LLM judge for real consent detection
        if kind == "tool" and payload.name in IRREVERSIBLE:
            if not consented:
                return False
            consented = False           # consent covers one irreversible action
    return True

# ---- Outcome grader: diff the database against s0 and the expected changes ----

def end_state_check(s0: dict, final: dict, expected_changes: dict):
    """Every expected key has the expected value, and nothing else changed."""
    wrong = {k: final.get(k) for k, v in expected_changes.items() if final.get(k) != v}
    changed = {k for k in s0.keys() | final.keys() if s0.get(k) != final.get(k)}
    collateral = changed - expected_changes.keys()
    return {"pass": not wrong and not collateral, "wrong": wrong, "collateral": collateral}

# ---- Apply them to the trial from the diagram ----

expected = [call("get_reservation", id="R7K2"),
            call("search_flights", origin="JFK", dest="CDG", date="2026-10-09"),
            call("update_reservation", id="R7K2", flight="AF125")]
actual = [expected[0], expected[1], expected[1], expected[2]]   # one repeated search

print(precision_recall(actual, expected))   # (0.75, 1.0)
print(exact_match(actual, expected))        # False  <- too strict for this trial
print(in_order_match(actual, expected))     # True
print(any_order_match(actual, expected))    # True
print(redundant_calls(actual))              # 1

s0    = {"R7K2.flight": "AF123", "R7K2.cabin": "economy", "R7K3.flight": "BA7",  "payments": ()}
final = {"R7K2.flight": "AF125", "R7K2.cabin": "economy", "R7K3.flight": "BA7",  "payments": (80,)}
print(end_state_check(s0, final, {"R7K2.flight": "AF125", "payments": (80,)}))
# {'pass': True, 'wrong': {}, 'collateral': set()}

Read the outputs carefully, because they contain the whole lesson of this module in miniature. Exact match says fail, for a trial that did everything right except one harmless repeated search. Precision (0.75) captures the waste, recall (1.0) says nothing required was skipped, in-order match says the logical sequence was respected, and the end-state check says the world is exactly as it should be with no collateral damage. A sensible scorecard for this task would be:

Step 5: metrics over the whole suite

One trial proves little. The full eval runs, say, 60 tasks (a mix of straightforward changes, over-budget fares where the right answer is no change, cancellations the user did not ask for, impossible requests under policy, and different personas) with k = 4 trials each. You then report task success rate with a confidence interval (Module 04, clustered by task because trials of the same task are correlated), pass^4 for reliability, policy-violation rate (which should be 0, and any non-zero value is investigated trial by trial), mean cost and steps per task, and a breakdown of exit reasons. When you change the prompt or the model, you re-run the same tasks with the same simulator and compare with a paired test.

Putting it together: a checklist

1. Name the system under test: model, scaffold version, tool set, budgets. Treat all of them as part of the result.
2. Build a resettable, isolated, deterministic environment, and measure its own flakiness before measuring the agent.
3. Write outcome checks first, as a full diff against s0, including "nothing else changed".
4. Add process checks as constraints: policy, consent, forbidden actions, loop detection. Avoid demanding a specific route.
5. Grade the final response against the end state, not just for style.
6. Use a simulated user only when the task needs one, pin it, and audit it.
7. Run k trials per task, report success with intervals, pass^k, cost, steps and exit reasons.
8. Read transcripts. Every number above is a summary of trajectories; the trajectories are where the truth is.

Check yourself

Sources and further reading

• Anthropic, Building effective agents (Dec 2024): augmented LLM, workflows vs agents, sandboxed testing.
• Anthropic, Demystifying evals for AI agents (Jan 2026): task, trial, grader, transcript, outcome; outcomes over paths; isolation; pass@k vs pass^k.
• Yao et al., τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024), and Barres et al., τ²-Bench (2025); repo sierra-research/tau2-bench.
• Yang et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024).
• Anthropic, Raising the bar on SWE-bench Verified with Claude 3.5 Sonnet: scaffold dependence.
• SWE-bench leaderboards including the bash-only view: swebench.com.
• Merrill et al., Terminal-Bench (2026): model × harness results.
• Xie et al., OSWorld (2024) and OSWorld-Verified.
• Trivedi et al., AppWorld (2024): state-based tests with collateral-damage checks.
• Guo et al., StableToolBench (2024): virtual API server for reproducible tool evals.
• Kapoor et al., AI Agents That Matter (2024): cost-controlled agent evaluation.
• Google Cloud, Evaluate Gen AI agents (trajectory metrics), summarized in Mete Atamel, Gen AI Evaluation Service: Agent Metrics (2025).
• Google ADK, Why evaluate agents and evaluation criteria (tool_trajectory_avg_score match types, user simulation).
• Valmeekam et al., PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change (2022; NeurIPS 2023 Datasets and Benchmarks).
• Xie et al., TravelPlanner: A Benchmark for Real-World Planning with Language Agents (2024): hard vs commonsense constraints, micro and macro pass rates, final pass rate.
• Zheng et al., NATURAL PLAN: Benchmarking LLMs on Natural Language Planning (Google DeepMind, 2024); data at google-deepmind/natural-plan.
• Kambhampati et al., LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks (2024): the case for external plan verifiers.
• Chen et al., Evaluating Large Language Models Trained on Code (2021): pass@k.