Anatomy of an eval

Every eval, from a ten-line script to SWE-bench, is built from the same parts: a dataset of samples, a system under test, a grader, a metric, and a harness that runs it all and writes logs. Learn the parts and you can read, build or criticize any eval you meet.

The parts, at a glance

In Module 01 we defined an eval as a fixed set of inputs, run through your AI system, scored by a grader and aggregated into metrics. This module takes that sentence apart like a mechanic takes apart an engine. There are four parts you design, and two supporting parts that make the whole thing trustworthy:

The rest of this module goes through each part in order: first the sample (the atom of every dataset), then how to build a whole dataset, then the system under test, then how the harness runs everything, and finally code you can run.

What a sample looks like

A sample (also called a case, item, example, task instance or, in Anthropic's vocabulary, a task) is one row of the dataset. It is a self-contained test: everything the harness needs to set up one attempt and everything the grader needs to judge it. Most tools store samples as JSONL (JSON Lines: one JSON object per line), because it is easy to append to, diff in git, and stream.

A sample can have up to six kinds of field. Only input is mandatory; the rest depend on the task.

Example (a): a question-and-answer item

The simplest kind of sample: one question, one short answer, graded by comparing text. Here is a support bot's knowledge check. Notice that the target is not a single string: it lists facts that must appear and a fact that must not (the old policy), because the bot may phrase its answer many ways.

{"id": "ret-0142",
 "input": "Hi, I bought headphones 3 weeks ago and opened them. Can I still return them?",
 "target": {"must_include": ["30 days"], "must_not_include": ["14 days"]},
 "rubric": "Correct if the reply says opened electronics can be returned within 30 days of delivery. Incorrect if it refuses, invents a restocking fee, or quotes the old 14-day policy.",
 "metadata": {"category": "returns", "difficulty": "easy", "source": "prod-logs-2026-08", "policy_version": "v7"}}

(Real JSONL puts each object on a single line. It is wrapped here for reading.) The must_include list suits a cheap code grader; the rubric is there so a model grader can handle phrasings the code misses, such as "a month" or "thirty days". Keeping both lets you compare the two graders against each other, which is how you catch grader bugs.

Example (b): a tool-using agent item

An agent that works with tools is graded mainly on what it did, not what it said. This item is modeled on τ-bench (tau-bench, from Sierra), where an LLM plays the customer from hidden instructions, the agent calls API tools under a written policy, and grading compares the final database with an annotated goal state.

{"id": "retail-0031",
 "input": {"user_instructions": "You are Dana Lee (user_id u_77). You want to cancel order #W4821 because you ordered the wrong size, and get the refund on your original card. You don't know the order total. Only give your zip code (94110) if asked."},
 "setup": {"db_fixture": "retail_db_v3.json", "policy_doc": "retail_policy_v3.md",
           "tools": ["find_user", "get_order", "cancel_pending_order", "transfer_to_human"]},
 "target": {"final_db_diff": {"orders.W4821.status": "cancelled",
                              "orders.W4821.refund_method": "original_payment"},
            "must_communicate": ["refund", "5-7 business days"],
            "forbidden_tool_calls": ["cancel_pending_order(order_id != 'W4821')"]},
 "metadata": {"domain": "retail", "difficulty": "medium", "turns_expected": "4-8",
              "author": "mp", "added": "2026-09-01"}}

Three things are new compared with Q&A. First, the input is instructions for a simulated user, not a message for the agent: the agent must discover the details through conversation. Second, setup names a database fixture that the harness loads fresh for every attempt, so one run's cancellations cannot leak into the next. Third, the target is a state (what the database must look like afterwards) plus checks on the conversation, rather than a string. There are many valid conversations; there is exactly one correct end state. Module 06 goes deep on this style.

Example (c): a coding-agent item with a hidden test

For coding agents, the gold standard is to grade by running tests. SWE-bench does this with real GitHub issues: the agent edits the repository, then the harness runs FAIL_TO_PASS tests (tests that fail before the fix and must pass after) and PASS_TO_PASS tests (tests that must keep passing, so the agent did not break anything). The key design trick is that the tests that check the fix are hidden: they are copied into the environment only after the agent has finished.

{"id": "dates-0007",
 "input": "Issue #212: parse_date('2024-02-30') returns 2024-03-01 instead of raising ValueError. Invalid days should raise. Please fix it in the repo at /workspace.",
 "setup": {"image": "python:3.12-slim", "repo": "acme/dateutils", "commit": "3f2a9c1",
           "script": "pip install -e /workspace"},
 "target": {"hidden_test_file": "tests/test_invalid_days.py",
            "fail_to_pass": ["tests/test_invalid_days.py::test_feb_30_raises",
                             "tests/test_invalid_days.py::test_apr_31_raises"],
            "pass_to_pass": ["tests/test_parse.py", "tests/test_leap_years.py"]},
 "metadata": {"language": "python", "difficulty": "easy", "files_touched_in_gold_patch": 1,
              "gold_patch": "patches/dates-0007.diff"}}

Why hide the test? If the agent can read test_invalid_days.py, it can write code that satisfies exactly those two asserts (for example, special-casing February 30th) instead of fixing the real bug. Hiding the test means the agent has to understand the issue, the way a human developer would. The pass_to_pass list is just as important: without it, an agent that deletes half the parser to make the new test pass would score perfectly. Module 07 covers this in depth, including the documented problem that tests can also be too strict.

Where datasets come from

Samples have to come from somewhere. There are four main sources, and mature teams use all four, each for what it is good at.

Synthetic data done well

Naively asking a model "write 100 customer questions" gives 100 polite, similar questions. A better method is to define dimensions first, then generate combinations. For a support bot: intent (refund, tracking, account, complaint), persona (new user, angry repeat customer, non-native speaker), difficulty (one fact, needs two tools, policy exception), channel quirks (typos, very long message, two questions at once). Generate a few inputs per combination, discard duplicates, and have a human write or verify the target for each. The dimensions guarantee spread; the human check guarantees correctness.

How many samples do you need?

This is the most common beginner question, and the honest answer is "enough that the differences you care about are bigger than the noise". Here is the intuition; Module 04 does the maths properly.

Imagine the true pass rate of your agent is 70%. You run it on n samples. Just by luck of which samples you picked, the measured rate wobbles around 70%. The size of that wobble is the standard error, roughly sqrt(p × (1 − p) / n), and a 95% confidence interval is about 1.96 standard errors either side. The wobble shrinks with the square root of n: to halve it, you need four times as many samples.

Three practical consequences:

• Small sets are for finding failures, not measuring rates. Twenty well-chosen samples are excellent for noticing that your agent always forgets to confirm before canceling. They cannot tell you whether prompt B is 3 points better than prompt A.
• Run agents several times per sample. Agents are stochastic: the same sample can pass once and fail the next time. Running each sample for several epochs (repeats) reduces the noise from sampling and lets you measure consistency (pass^k, Module 04). Repeats do not replace more samples: they reduce noise within a sample, not the luck of which samples you chose.
• Grouped samples count for less. If ten questions all come from the same document or the same repository, they are not ten independent pieces of evidence. Anthropic's Evan Miller showed naive standard errors can understate uncertainty by more than 3× in such cases (use clustered standard errors).

Coverage: testing what matters, not just what's easy

Two datasets of 200 samples can have very different value. One is 200 variations of "where is my order?". The other deliberately spreads samples across every kind of request, every level of difficulty, and the nasty cases. That spread is called coverage, and the best tool for thinking about it is a coverage matrix: put your main categories on one axis, difficulty on the other, and count samples in each cell.

Things to cover deliberately:

• Categories: every major intent or task type, roughly in proportion to how often users hit it, plus extra samples for rare but costly ones (a wrong refund costs more than a clumsy greeting).
• Difficulty: easy items catch regressions; hard items tell you about capability and give headroom for improvement.
• Edge cases: empty input, very long input, two requests in one message, the user changes their mind halfway, a tool returns an error, the order does not exist.
• Negative cases: situations where the right move is to refuse, ask a clarifying question, or hand off to a human. Datasets full of "do X" tasks train you to build agents that always do something.
• Adversarial cases: prompt injection in a retrieved document, users pressuring the agent to break policy, jailbreak attempts. Module 17 covers these.

Always report scores per slice as well as overall. A single average can hide a disaster: if billing improves by 5 points and account questions collapse by 30, the average may barely move.

Golden sets and capability sets

Module 01 introduced the purpose axis: regression versus capability. In dataset terms this means keeping two kinds of set, which behave very differently.

Mixing the two into one number muddles both signals. A regression hidden in 50 golden items gets diluted by 500 hard capability items, and progress on the hard items gets diluted by the easy ones that were always at 100%. Keep them as separate tasks with separate dashboards. A benchmark where every system scores near the top is saturated: it has stopped being a capability set and can only serve as a regression check.

Dev, test, and contamination

Here is a trap that catches almost everyone. You build an eval, look at the failures, tweak the prompt, re-run, look at the new failures, tweak again. After fifty rounds your score has gone from 60% to 90%. But you have been fitting the prompt to these exact samples, the same way a student who memorizes last year's exam paper does well on that paper and not necessarily on the next. The 90% is optimistic.

The fix comes from machine learning: split your data.

• Dev set (development set): the samples you look at, debug on, and tune against. Look as much as you like.
• Test set (held-out set): samples you run only occasionally, for example before shipping or when choosing between two finalists, and whose individual failures you do not use to tweak the system. If you do start tuning on it, it quietly becomes a second dev set and you need fresh held-out data.
• In practice many small teams skip the split early on, because they have only 50 samples and need every one for finding failures. That is reasonable, as long as you know the score is optimistic, and you start holding samples back once you have a few hundred.

Contamination (or data leakage) is the model-level version of the same problem: the test data, or its answers, ended up in the model's training data. Public benchmarks are especially exposed. In February 2026 OpenAI stopped reporting SWE-bench Verified, partly because every frontier model it tested could reproduce some of the gold patches verbatim. Your private eval is safer, but not immune: if you paste samples into public issue trackers or share them with vendors, they can leak. Defenses include keeping test sets private, embedding a canary string (a unique random marker that lets model builders filter the data out of training and lets you test whether a model has seen it), and regularly adding fresh samples. Module 18 covers contamination in depth.

The system under test

People say "we evaluated Claude" or "we evaluated GPT", but that is almost never what happened. What was evaluated is a whole system, and every layer of it can change the score.

The eval run lifecycle

A harness runs an eval in a fixed sequence of stages. Knowing them helps you debug: every weird score can be traced to one stage.

1. Load. Read the dataset, check every sample has the required fields, apply any filters (a slice, a limit for a quick smoke test), and shuffle if needed with a fixed seed.
2. Set up the environment. For a plain Q&A eval this does nothing. For an agent it may start a Docker container, check out a git commit, load a database fixture, or launch a browser. It must be fresh each time: state left over from a previous attempt is one of the most common hidden bugs.
3. Run. Call the system under test. For an agent this is a loop of model calls and tool calls, bounded by limits (maximum messages, tokens, time or cost) so a stuck agent cannot run forever. The harness also handles rate limits and retries on API errors.
4. Grade. Apply the grader to the output and, for agents, the final environment state and the transcript. For hidden tests, this is when they are copied in and run.
5. Aggregate. Combine per-sample scores into metrics: overall mean, standard error, per-slice breakdowns, pass@k or pass^k if there were several epochs.
6. Log. Save the configuration, every transcript, every score and grader explanation, token use, cost, timing and errors.
7. Compare. Put the result next to the baseline (last release, other model) on the same samples, look at which samples changed from pass to fail and vice versa, and read those transcripts.

A minimal eval from scratch

Frameworks are helpful, but it is worth building one eval with nothing but Python first, so you can see there is no magic. This eval tests a support assistant on policy questions. It uses Anthropic's Python SDK; any LLM API works the same way.

First, the dataset, support_qa.jsonl (three lines shown; a real one would have dozens to hundreds):

{"id": "ret-0142", "input": "I opened my headphones 3 weeks ago. Can I still return them?", "target": {"must_include": ["30 days"], "must_not_include": ["14 days"]}, "metadata": {"category": "returns", "difficulty": "easy"}}
{"id": "shp-0007", "input": "How much is express shipping to Canada?", "target": {"must_include": ["$25"]}, "metadata": {"category": "shipping", "difficulty": "easy"}}
{"id": "acc-0019", "input": "Can I merge two accounts into one?", "target": {"must_include": ["cannot"], "must_not_include": ["merged"]}, "metadata": {"category": "account", "difficulty": "medium"}}

Then the eval itself. Every one of the five parts is labeled.

"""A complete eval in ~40 lines: dataset, system under test, grader, metric, log."""
import json, math, re, time
from collections import defaultdict
import anthropic

# ---- The system under test: ALL of this is part of what you measure ----
MODEL = "claude-sonnet-5-5"      # pin an exact model id you have access to
EFFORT = "low"                   # thinking effort is a setting too: record it
SYSTEM = open("prompts/support_v7.txt").read()   # system prompt, versioned in git
client = anthropic.Anthropic()   # reads ANTHROPIC_API_KEY from the environment

def system_under_test(question: str) -> str:
    resp = client.messages.create(model=MODEL, max_tokens=4000,
                                  output_config={"effort": EFFORT},
                                  system=SYSTEM, messages=[{"role": "user", "content": question}])
    # current Claude models think first, so take the text block, not content[0]
    return "".join(b.text for b in resp.content if b.type == "text")

# ---- 1. Dataset ----
samples = [json.loads(line) for line in open("support_qa.jsonl") if line.strip()]

# ---- 2. Grader: required facts present, forbidden facts absent ----
def norm(s: str) -> str:
    return re.sub(r"[^a-z0-9$ ]", "", s.lower())

def grade(output: str, target: dict) -> bool:
    out = norm(output)
    has_all = all(norm(x) in out for x in target["must_include"])
    has_bad = any(norm(x) in out for x in target.get("must_not_include", []))
    return has_all and not has_bad

# ---- 3. Run + grade every sample ----
results = []
for s in samples:
    out = system_under_test(s["input"])          # note: the target is NOT passed in
    results.append({"id": s["id"], "category": s["metadata"]["category"],
                    "output": out, "passed": grade(out, s["target"])})

# ---- 4. Metric: accuracy with a 95% confidence interval, overall and per slice ----
n = len(results)
p = sum(r["passed"] for r in results) / n
print(f"accuracy {p:.1%} ± {1.96 * math.sqrt(p * (1 - p) / n):.1%}  (n={n})")
by_cat = defaultdict(list)
for r in results:
    by_cat[r["category"]].append(r["passed"])
for cat, v in sorted(by_cat.items()):
    print(f"  {cat:10s} {sum(v)}/{len(v)}")

# ---- 5. Log: config + every output, so you can read transcripts later ----
with open(f"logs/run_{int(time.time())}.jsonl", "w") as f:
    f.write(json.dumps({"model": MODEL, "effort": EFFORT, "system": SYSTEM}) + "\n")
    for r in results:
        f.write(json.dumps(r) + "\n")

Run it and you get something like accuracy 78.0% ± 8.1% (n=100) followed by per-category counts. That is a real eval. It has all the parts: fixed dataset, a clearly defined system, a deterministic grader, a metric with uncertainty, per-slice results, and a log of every output.

It also has real weaknesses, and seeing them is the point:

• The grader is naive. "Returns are accepted within 30 days, but not after 14 days of use" contains the forbidden "14 days" and fails, even if it is correct. "Thirty days" contains no "30 days" and fails. Substring graders are cheap and useful, but you must check them against human judgment (Module 03).
• One attempt per sample. Model outputs are sampled, so the same question can pass once and fail the next time. Add an epochs loop.
• No error handling. If the API times out, the script crashes. A real harness retries and records errors separately from failures.
• Sequential. 100 samples run one after another. Real harnesses run many in parallel within rate limits.
• Only single-turn. No tools, no environment, no agent loop.

You could fix all of these yourself. Or you could use a framework that already has.

The same eval in Inspect

Inspect (inspect_ai) is an open-source eval framework from the UK AI Security Institute, now co-developed with the non-profit Meridian Labs, and used by several government AI safety institutes. Its vocabulary maps directly onto the anatomy in this module:

# support_qa.py  (checked against inspect_ai 0.3.x)
import re
from inspect_ai import Task, task
from inspect_ai.dataset import Sample, json_dataset
from inspect_ai.model import GenerateConfig
from inspect_ai.scorer import CORRECT, INCORRECT, Score, Target, accuracy, scorer, stderr
from inspect_ai.solver import TaskState, generate, system_message

SYSTEM = open("prompts/support_v7.txt").read()

def norm(s: str) -> str:
    return re.sub(r"[^a-z0-9$ ]", "", s.lower())

# Dataset: map our JSONL fields onto Inspect's Sample
def record_to_sample(r: dict) -> Sample:
    return Sample(
        id=r["id"],
        input=r["input"],
        target=r["target"]["must_include"],            # Target can be a list of strings
        metadata={**r["metadata"],
                  "must_not_include": r["target"].get("must_not_include", [])},
    )

# Grader + metric: the same logic as before, packaged as a Scorer
@scorer(metrics=[accuracy(), stderr()])
def facts_present():
    async def score(state: TaskState, target: Target) -> Score:
        out = norm(state.output.completion)
        has_all = all(norm(t) in out for t in target.target)
        has_bad = any(norm(b) in out for b in state.metadata["must_not_include"])
        return Score(value=CORRECT if has_all and not has_bad else INCORRECT,
                     answer=state.output.completion)
    return score

@task
def support_qa():
    return Task(
        dataset=json_dataset("support_qa.jsonl", record_to_sample),
        solver=[system_message(SYSTEM), generate()],     # the system under test (minus the model)
        scorer=facts_present(),
        config=GenerateConfig(max_tokens=4000, reasoning_effort="low"),
        epochs=3,                                        # run every sample 3 times
    )

# the model is chosen at run time, so you can compare models on the same Task
inspect eval support_qa.py --model anthropic/claude-sonnet-5-5
inspect eval support_qa.py --model openai/gpt-4.1 --limit 10   # quick smoke test
inspect view                                                    # browse logs and transcripts

What you gained for free: parallel execution with rate-limit handling, retries, epochs, a standard error, a log file for every run containing every message, and a web viewer for transcripts. And because the model is a command-line argument, comparing models is a one-word change.

Scaling up: the coding item from earlier

The same four concepts stretch to agents. For sample (c), the solver becomes a ReAct-style agent (a model that alternates between reasoning and tool calls) with a bash tool inside a Docker sandbox, and the scorer copies in the hidden tests after the agent has finished and runs them.

from inspect_ai import Task, task
from inspect_ai.agent import react
from inspect_ai.scorer import CORRECT, INCORRECT, Score, Target, accuracy, scorer
from inspect_ai.solver import TaskState
from inspect_ai.tool import bash, text_editor
from inspect_ai.util import sandbox

@scorer(metrics=[accuracy()])
def hidden_tests():
    async def score(state: TaskState, target: Target) -> Score:
        # The hidden test file enters the container only NOW, after the agent is done.
        await sandbox().write_file("/workspace/" + state.metadata["hidden_test_file"],
                                   state.metadata["hidden_test_source"])
        tests = state.metadata["fail_to_pass"] + state.metadata["pass_to_pass"]
        result = await sandbox().exec(["python", "-m", "pytest", "-q", *tests],
                                      cwd="/workspace", timeout=600)
        return Score(value=CORRECT if result.returncode == 0 else INCORRECT,
                     explanation=result.stdout[-3000:])   # keep the test output for debugging
    return score

@task
def date_bugs():
    return Task(
        dataset=load_coding_samples("coding_tasks.jsonl"),  # your loader: builds Samples with metadata
        solver=react(tools=[bash(timeout=180), text_editor()]),
        scorer=hidden_tests(),
        sandbox="docker",          # a fresh container per sample (compose.yaml defines the image)
        message_limit=60,          # stop runaway agents
    )

Compare this with the Q&A task: the shape is identical (dataset, solver, scorer, metrics, limits), only the pieces are richer. That is the point of learning the anatomy.

Reading logs and transcripts

An eval produces a number, but the number is the least informative thing it produces. The transcripts (the full record of each attempt: every message, tool call, tool result and grader explanation) are where you learn what is actually going on. Experienced practitioners are close to unanimous on this: before you trust a score, read a sample of transcripts, especially failures, and some passes too.

Transcripts reveal problems no metric can:

• Grader bugs. Passes that should be failures (as above) and failures that are actually correct answers the grader could not parse.
• Broken samples. A task that is impossible because the fixture is missing a record, or ambiguous because the question has two reasonable readings. If a sample fails for every model you try, suspect the sample before the models.
• Infrastructure failures. A container that failed to install a package, a rate-limit error that the agent treated as a tool result, a browser page that never loaded.
• Shortcuts and cheating. An agent that edits the tests instead of the code, reads the answer from a file it should not have access to, or hard-codes the expected output. Labs have documented agents doing exactly this; Module 18 has examples.
• The kind of failure. "Failed 22%" is not actionable. "Half the failures are the agent not asking for the order number" is. This is error analysis (Module 12).

In Inspect, inspect view gives a browsable interface. You can also read logs in code, which is handy for finding the samples that flipped between two runs:

from inspect_ai.log import read_eval_log

log = read_eval_log("logs/2026-09-29T10-02-11_support-qa_Ab3x.eval")
print(log.eval.model, log.results.scores[0].metrics["accuracy"].value)

for s in log.samples:                        # one entry per sample × epoch
    sc = s.scores["facts_present"]
    if sc.value == "I":                      # "I" = INCORRECT, "C" = CORRECT
        print(f"--- {s.id} (epoch {s.epoch}) ---")
        for m in s.messages:                 # the full conversation
            print(m.role, ":", m.text[:200])

Worked example: from "be helpful" to samples

Real eval projects rarely start with a crisp spec. They start with a wish: "the support bot should be helpful." You cannot grade "helpful". Here is how to turn it into something you can.

Step 1: ask "helpful how?" until the answers are observable

Interview the people who own the product, and read 50 to 100 real conversations (or, before launch, role-play them). Ask: when a conversation went well, what did the bot do? When it went badly, what went wrong? You will get answers like "it told the customer the wrong return window", "it promised a refund it cannot give", "it wrote three paragraphs for a yes/no question". Each complaint is a criterion in disguise. Anthropic's guidance puts it as success criteria that are specific and measurable: not "good answers" but "states the correct policy for the customer's situation".

Step 2: write criteria a stranger could apply

The test for a good criterion: if you gave it to two different people (or a person and a model), would they agree on almost every case? "Friendly" fails that test. "Does not blame the customer, and apologizes at most once" passes.

Step 3: write samples that exercise each criterion

For each criterion, write samples that should pass and samples designed to tempt a failure. Tempting a failure is the valuable part: "I was charged twice, refund me NOW" tests whether the bot caves to pressure and promises a refund. Tag every sample with its criterion, category and difficulty in metadata so you can build the coverage matrix.

{"id": "handoff-0004", "input": "I was charged twice for order #5530. Refund me NOW or I'm calling my bank.", "setup": {"db_fixture": "billing_dupe_charge.json"}, "target": {"required_tool_calls": [{"name": "transfer_to_human", "args": {"queue": "billing"}}], "forbidden_phrases": ["I have refunded", "refund has been issued"]}, "rubric": "Pass if the bot acknowledges the problem, does not promise or claim a refund, and transfers to the billing queue.", "metadata": {"criterion": "handoff", "category": "billing", "difficulty": "hard", "adversarial": "pressure"}}

Step 4: decide how criteria combine

Decide up front whether a sample passes only when all hard criteria pass (usual for correctness and safety) and report style scores separately, or whether you use a weighted score. Do not average a safety failure with a tone score: a polite answer that promises an unauthorised refund is a failure, not a 75%. Report each criterion as its own metric as well as the combined pass rate.

Step 5: check the eval before trusting it

Label 30 to 50 outputs by hand, run your graders on the same outputs, and compare. Where they disagree, fix the grader or the rubric. Also run a known-good reference answer through each grader: it must pass. And run a known-bad one: it must fail. Only then start using the numbers to make decisions.

Common anatomy mistakes

Almost every broken eval fails in one of a few ways. Learn to spot them in your own work and in published benchmarks.

Check yourself

Sources and further reading

• UK AI Security Institute and Meridian Labs, Inspect documentation: datasets, scorers, ReAct agent, sandboxing, eval logs. Code in this module checked against inspect_ai 0.3.x.
• Anthropic, Demystifying evals for AI agents (January 2026): task, trial, grader, transcript, outcome, harness.
• Anthropic, Define success criteria and build evaluations.
• OpenAI, Evaluation best practices.
• Evan Miller (Anthropic), Adding Error Bars to Evals (2024), and the blog summary.
• Jimenez et al., SWE-bench (FAIL_TO_PASS and PASS_TO_PASS tests); OpenAI, Introducing SWE-bench Verified (2024) and Why SWE-bench Verified no longer measures frontier coding capabilities (February 2026).
• Yao et al. (Sierra), τ-bench: A Benchmark for Tool-Agent-User Interaction (2024); task fixes in the tau2-bench repository.
• Hamel Husain, Your AI Product Needs Evals.
• Srivastava et al., BIG-bench (2022), which popularised benchmark canary strings.