Error analysis: building your own eval

Nobody hands you an eval for your agent. You build it by reading what your agent actually does, naming the ways it fails, counting them, and only then writing graders for the failures that matter. This module is the practitioner's playbook for going from zero to a working eval suite.

Why you start by looking at data

Most people who set out to "add evals" start in the wrong place. They open a tool, browse its list of built-in metrics (helpfulness, coherence, faithfulness, toxicity), pick five that sound sensible, and wire them up. A dashboard appears with numbers on it. The numbers move a little each week. Nothing about the product gets better.

The reason is simple: those metrics were written by someone who has never seen your agent fail. A generic "helpfulness" score cannot tell you that your refund agent keeps refunding the wrong order when a customer has three recent purchases, because that failure is specific to your product, your tools and your users. You only discover it by reading transcripts.

Hamel Husain and Shreya Shankar, who teach the most widely followed practitioner course on this, put the point bluntly. In their Evals FAQ they report spending "60-80% of our development time on error analysis and evaluation", and they advise against generic, off-the-shelf metrics as a starting point. Hamel's earlier essay Your AI Product Needs Evals says "You must remove all friction from the process of looking at data." The method they teach is called error analysis, and it is borrowed from qualitative research in the social sciences.

Two new terms before we go on:

• A trace (also transcript or trajectory) is the full record of one run of your agent: the user's messages, the model's replies, every tool call with its arguments and results, and the final state. (Module 01 introduced the vocabulary.)
• A failure mode is a named, recurring way the agent goes wrong, such as "refunds an order outside the 30-day window". Error analysis is the process of discovering your failure modes and how often each occurs.

The whole process at a glance

Here is the entire loop you will learn in this module. Each step gets its own section below, and the worked example at the end runs through all of them on one agent.

Notice the order. Evaluators appear in step 5, not step 1. You write graders only for failure modes you have actually observed and judged worth measuring. This is the single biggest difference between teams whose evals drive improvement and teams whose evals are decoration.

Step 1: collect traces

You need a pile of traces to read. There are two sources, and you usually want both.

Synthetic data done right: dimensions, tuples, then language

The naive approach is to ask an LLM: "Generate 100 questions a customer might ask a refund bot." You will get 100 polite, well-formed, near-duplicate questions that all hit the happy path. LLMs regress to the most typical output, so unstructured generation collapses toward the middle of the distribution, which is exactly where your agent already works.

The approach Husain and Shankar recommend has three stages:

1. Define dimensions. A dimension is an axis along which real inputs vary and along which you suspect your agent might fail. For a support agent: persona (calm, frustrated, confused, non-native speaker), feature or intent (full refund, partial refund, exchange, cancel), scenario (single order, multiple orders, outside return window, high-value order, missing order number). Choose dimensions from hypotheses about where things break.
2. Generate tuples. A tuple is one combination of dimension values, such as (frustrated, partial refund, multiple orders). You can enumerate all combinations with code, or ask an LLM to propose realistic ones, then filter out the nonsensical ones by hand. Structured tuples guarantee coverage of the corners.
3. Turn each tuple into natural language in a separate step. Now the LLM's only job is to write a realistic message for a specific, pinned-down situation. Separating "what situation" from "how would a person phrase it" is what gives you diversity without losing control.

That last point in the caption is a hidden superpower: because each synthetic input remembers its tuple, you can later ask "what is our failure rate for frustrated customers with multiple orders?" and get a real answer.

# Structured synthetic input generation: dimensions -> tuples -> natural language
import itertools, json, random
import anthropic

DIMENSIONS = {
    "persona":  ["calm", "frustrated", "confused non-native speaker", "terse"],
    "intent":   ["full refund", "partial refund", "exchange", "cancel order"],
    "scenario": ["single recent order", "3 orders in the last month",
                 "order is 45 days old (policy window is 30)",
                 "order worth $900 (needs human approval)",
                 "customer has no order number"],
}

# 1) Enumerate every combination, then drop ones that make no sense.
all_tuples = [dict(zip(DIMENSIONS, combo))
              for combo in itertools.product(*DIMENSIONS.values())]
def plausible(t):
    return not (t["intent"] == "exchange" and "no order number" in t["scenario"])
tuples = [t for t in all_tuples if plausible(t)]

# 2) Sample, over-weighting scenarios you believe are risky.
random.seed(7)
risky = [t for t in tuples if "45 days" in t["scenario"] or "$900" in t["scenario"]]
sample = random.sample(tuples, 30) + random.sample(risky, 15)

# 3) Separate call: turn each tuple into a realistic first message.
client = anthropic.Anthropic()          # reads ANTHROPIC_API_KEY
MODEL = "claude-sonnet-5-5"             # example id; use any current model

def to_message(t):
    prompt = (
        "Write the FIRST chat message a real customer would send to an online "
        "store's support bot. Do not mention these labels explicitly.\n"
        f"Persona: {t['persona']}\nWhat they want: {t['intent']}\n"
        f"Situation: {t['scenario']}\n"
        "Match the persona's tone, typos and length. Output only the message."
    )
    resp = client.messages.create(model=MODEL, max_tokens=200,
                                  messages=[{"role": "user", "content": prompt}])
    return resp.content[0].text.strip()

with open("synthetic_inputs.jsonl", "w") as f:
    for i, t in enumerate(sample):
        f.write(json.dumps({"id": f"syn-{i:03d}", "tuple": t,
                            "message": to_message(t)}) + "\n")

Step 2: open coding

Open coding is a term from qualitative research (grounded theory). It means reading raw material and writing short, free-form notes about what you observe, without a predefined list of categories. In our setting: you open a trace, read it, decide pass or fail, and if it fails, write one or two sentences describing what went wrong.

Rules that make open coding work:

• Note the first failure in each trace. Agents cascade: if the agent picked the wrong order in step 2, everything after is wrong too. The Evals FAQ advises focusing on "the first failure observed in a trace, as upstream errors can cause downstream issues." Fixing the root often removes the downstream symptoms for free.
• Be concrete, not abstract. "Refunded order #1182 but customer asked about the blue jacket (#1190)" is useful. "Bad accuracy" is not.
• Use a binary pass/fail verdict, not a 1-5 score. Husain and Shankar argue that a Likert scale mostly adds noise and hides the decision; a written critique carries the nuance instead.
• Do the first batch yourself. Their guidance is to review about 100 diverse traces and annotate at least the first 30 yourself before accepting any LLM-suggested codes. You are training your own judgment, and that cannot be delegated.
• Do not fix things mid-session. Write it down and keep reading. You want the distribution first.

Step 3: axial coding (the taxonomy)

Axial coding is the second qualitative pass: you take your pile of open codes and group them into categories, building connections ("axes") between notes that describe the same underlying problem. The output is a failure taxonomy: a short list of named failure modes, each with a clear definition and examples. The FAQ calls this "the most important step".

Good taxonomy categories share four properties:

1. Actionable: each category suggests a kind of fix. "Wrong order selected" points at order disambiguation. "Bad answer" points nowhere.
2. Mutually exclusive enough that one trace's first failure lands in one bucket. Overlap is fine at the margins, but if half your traces could go in either of two buckets, merge or redefine them.
3. Defined in writing, with a one-line definition and a pass and fail example each. That definition becomes the spec for any judge you build later.
4. Small: typically somewhere between 5 and 15 categories. If you have 40, you are still at the open-code level.

You can use an LLM to help here: paste your open codes and ask it to propose groupings. That is a reasonable accelerator after you have done enough coding yourself to judge its suggestions. Then relabel every trace with the final categories (a spreadsheet column is enough).

Step 4: count and prioritize

Once each failing trace has a category, count. This is the moment qualitative work becomes quantitative, and it is often the most clarifying moment of the whole process: the failure you were emotionally sure was the big problem is frequently third or fourth on the list.

Frequency is not the only input. Multiply mentally by severity (a wrongly issued $900 refund costs real money; a slightly stiff tone does not) and by ease of fix. A useful rule: the top of your list is whatever is frequent and costly and not trivially fixed by a prompt change.

When to stop reading: theoretical saturation

How many traces is enough? Qualitative researchers use the idea of theoretical saturation: you stop when new material stops producing new categories. Husain and Shankar translate this directly: "Continue until new traces stop revealing failure modes or changing existing ones," and they recommend reviewing at least 100 traces as a practical floor. Treat 100 as a rule of thumb, not a law. A narrow agent may saturate sooner; a broad general assistant can keep surprising you for much longer.

Who decides what "good" means

Error analysis depends on a judgment call on every trace: pass or fail? If three people with different mental standards annotate, you get three incompatible taxonomies and noisy labels. The Evals FAQ's answer is a benevolent dictator: "Appointing a single domain expert as a benevolent dictator is the most effective approach. This person becomes the definitive voice on quality standards."

• It should be a domain expert: for a refund agent, someone who knows the refund policy and what great support feels like (a support team lead), not necessarily an engineer. For a legal agent, a lawyer. Hamel's LLM-judge essay calls this role the principal domain expert.
• "Dictator" does not mean they work alone. Others can annotate and raise disagreements, but one person resolves them, so the labels stay consistent.
• When you do use multiple annotators (for example, to scale up), measure their agreement (Cohen's kappa, see Module 13) and resolve disagreements into sharper written definitions.

Build a custom review interface (it pays for itself)

If looking at data is the core activity, the tool you look at it with matters. Generic trace viewers show JSON; your domain expert does not want to read JSON. The FAQ recommends building a simple custom annotation app, and lists features that matter: render the trace the way the domain looks ("If you're evaluating generated emails, render them to look like emails"), show progress ("Trace 45 of 100"), keyboard hotkeys for pass, fail and next, filtering and search by metadata, and surfacing traces already flagged by guardrails or automated evaluators. Keep all the context on one screen.

Why does it pay off? Because the cost of error analysis is almost entirely reviewer time and attention. A reviewer who has to click into nested JSON, cross-reference the order database in another tab, and type notes into a separate spreadsheet will review a fraction as many traces and will quit sooner. Today, a coding agent can build a bespoke review app for your trace format in an afternoon, which makes the trade even easier.

Step 5: decide, fix directly or build an evaluator

Now you have a ranked list of failure modes. For each one, the question is not "which metric?" but "what is the cheapest thing that makes this stop happening, and do I need to keep watching it?"

Three principles guide this step:

• Fix obvious bugs first. If the system prompt never mentions the 30-day window, the agent is not "failing", it was never told. Fix it. Husain and Shankar advise building LLM judges only for failures that persist after such fixes; a judge for a problem you could fix in one line is wasted work.
• Prefer code checks. For agents, a surprising amount is objectively checkable from the environment and the tool log: was issue_refund called on an order older than 30 days? Was escalate_to_human called for amounts over $500? Did the final message claim success when the refund API returned an error? These are cheap, deterministic, and never drift (see Module 03).
• One judge per failure mode. When you do need an LLM judge ("did the agent promise anything not in the policy?"), make it binary and narrow. The FAQ names "directing their LLM judge to catch too many different kinds of errors at once" as the most common mistake. Your taxonomy definition, with its pass and fail examples, is the judge's rubric.

Steps 6 to 8: validate, automate, iterate

Validate judges. An LLM judge is a classifier, and classifiers make mistakes. Before you trust it, compare its verdicts against your domain expert's labels on traces the judge was not tuned on. The FAQ recommends train, dev and test splits of your labeled traces and reporting the judge's true positive rate (TPR: of the traces the human passed, how many the judge passed) and true negative rate (TNR: of the traces the human failed, how many the judge failed). Your error analysis labels are exactly the ground truth you need. Module 13 does this in depth, including how to correct the measured pass rate for judge error.

Add to CI and production. Once checks and validated judges exist, run them automatically: on a fixed regression set on every prompt or code change, and on a sample of production traffic. Failures in production feed straight back into step 1. Module 15 covers the plumbing.

Iterate. Keep a steady trickle of fresh traces flowing into review. The FAQ notes you should expect pass rates well below 100% on a healthy eval (they suggest around 70% as a sign it is challenging enough; a perfect score suggests the eval is too easy). And be ready for your definitions to change, which is the subject of the next section.

Criteria drift: your standards change as you grade

Shreya Shankar and colleagues studied people building LLM evaluators in the paper Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences (UIST 2024). They built a tool, EvalGen, that proposes evaluation criteria and candidate assertion implementations and aligns them with a user's grades. Their key observation is criteria drift: "users need criteria to grade outputs, but grading outputs helps users define criteria."

What this means in practice:

• Do not try to write the perfect rubric up front. Write a first version, grade, and expect to revise.
• When a definition changes, re-label earlier traces under the new definition, or your counts mix two standards.
• Keep your taxonomy and judge rubrics in version control, with the date of each change, so you can explain why a metric jumped.
• Drift is also a reason to keep humans in the loop long after launch: a judge frozen with version-1 criteria silently becomes misaligned with what your expert now considers a failure.

Agent-specific error analysis

Everything above works for single-turn LLM features. Agents add length, branching and side effects, which calls for a few extra tools. The Evals FAQ's entry on agentic workflows suggests two phases: first treat the agent as a black box and judge end-to-end task success (did it meet the user's goal?), then, for failing workflows, do step-level diagnostics: tool choice, parameter extraction, error handling, context retention, efficiency, and goal checkpoints.

Transition failure matrices: where does it break?

An agent moves through states (steps or phases, such as "classify intent", "look up order", "check policy", "issue refund"). For each failing trace, record two things: the last state that succeeded and the first state that failed. Tally these pairs in a grid with rows for the last successful state and columns for the first failure. That grid is a transition failure matrix, a technique from Husain and Shankar's agentic-workflow FAQ entry (their example shows 12 failures clustered on a GenSQL to ExecSQL transition, instantly revealing where to look). Bright cells show where your agent's failures cluster.

Why this is powerful: a flat list of failure categories tells you what went wrong; the matrix tells you where in the workflow it went wrong, which maps directly onto which prompt, tool or sub-agent to change. It also works beautifully with code checks: once you have a per-step success check, you can compute the matrix automatically over thousands of traces instead of 120.

Tool error, reasoning error, or environment error?

When an agent step fails, it helps to ask whose fault it was, because each kind of fault has a different owner and a different fix. The classification below is a practical convention used widely in agent debugging (Anthropic's eval guidance makes the same distinction when it warns that a task nobody can solve usually means a broken task, not a weak agent).

Two subtleties. First, a tool error can cause a reasoning error: if lookup_orders returns dates as Unix timestamps, the model may misjudge the order's age. Record the root cause (the tool). Second, how the agent reacts to a genuine tool failure is itself a reasoning question: when the refund API times out, "tell the user it's done" is a reasoning failure even though the tool failed first. In the taxonomy that became "tool failure mishandled".

Reading long transcripts efficiently

Agent traces can run to hundreds of steps. Reading every token of 100 of them is not realistic. Techniques that help:

• Outcome first. Start at the end: check the final state and final message against the task. If the outcome is right, skim; if wrong, walk backward to find where it went off the rails. This follows the black-box-then-steps advice above.
• Render, do not dump. Collapse tool results by default, show tool name, arguments and a one-line result summary, and highlight errors, retries and very long gaps. This is your custom review UI again.
• Jump to anomalies. Sort or filter steps by tool errors, repeated identical calls (loops), unusually long outputs, and state changes. Many failures hide at the first error or the first repeated call.
• Let an LLM pre-read, but verify. A model can summarize a trace and propose where the first failure is. Treat that as a pointer, not a verdict, especially before you have coded 30 or so traces yourself. For LLM judges on long traces, the FAQ suggests giving the judge a tool to search the parts it needs rather than stuffing everything into the prompt.
• Look for what is missing. Anthropic's tool-writing guide notes that "what agents omit ... can often be more important than what they include". The refund agent that never called check_policy produces no error at all; you only see it if you notice the absence.

What Anthropic recommends for agent eval suites

Anthropic's engineering post Demystifying evals for AI agents (January 9, 2026) distills lessons from building evals for agents like Claude Code. Its advice lines up closely with error analysis:

Worked example: a refund agent, end to end

Let us run the whole process on one agent. ShopCo's refund agent (a made-up company) chats with customers and has five tools: lookup_orders, check_policy, issue_refund, escalate_to_human, and send_message. The policy: refunds within 30 days of delivery; refunds over $500 must be escalated to a human; refunds take up to 10 business days. All data below is invented for teaching.

1. Collect

We have two weeks of beta traffic (75 traces sampled, oversampling thumbs-down and escalations) plus 45 synthetic traces from the tuple generator above, with fixture orders generated from each tuple. Total: 120 traces.

2. Open coding: six sample traces

Here are six of the 120, as the reviewer saw them, with the open code they wrote. Notice each note describes the first failure only, concretely.

3. Axial coding: the taxonomy

After open-coding all 120 traces (45 failures), the team grouped the notes into nine failure modes. Each has a written definition that later becomes a check or a judge rubric.

4. Count

The overall pass rate is 75 / 120 = 62.5%. The top two modes (21 of 45 failures) are both about the policy and order lookup path, which the transition failure matrix earlier also pointed at: the Lookup to Policy cell (12) and the Classify to Lookup cell (7).

5. Decide

6 to 8. Validate, automate, iterate

The two LLM judges get validated: the expert's 120 labels (plus 60 more collected for the purpose) are split into train (few-shot examples), dev (tune the judge prompt) and test (report TPR and TNR once). The six code checks and two validated judges go into CI against a regression set built from the failing traces (following Anthropic's "20-50 tasks from real failures" advice, this suite starts at 45 tasks), balanced with passing cases where refunds should be approved and ones that should be denied. After the fixes, the team samples 100 new traces and reruns error analysis. A new mode appears, "refunds partial shipments at full price", and the loop begins again.

The code: from CSV of annotations to counts and a matrix

You do not need a platform for any of this. A CSV (or a spreadsheet exported to CSV) and pandas carry you a long way. Suppose your review UI writes one row per trace:

trace_id,source,persona,verdict,open_code,last_ok_state,first_failed_state
T-017,beta,frustrated,fail,"refunded mug not jacket; took latest order",Classify,Lookup
T-033,beta,calm,fail,"policy said ineligible (52d), refunded anyway",Lookup,Policy
T-048,synthetic,calm,fail,"promised free return label + 2-3 days",Refund,Respond
T-061,beta,terse,fail,"refund API timeout, told user it worked",Policy,Refund
T-072,beta,confused,pass,,,
T-090,synthetic,calm,fail,"$900 refund, needs_approval ignored",Policy,Escalate

Group open codes into axial codes and count

import pandas as pd

df = pd.read_csv("annotations.csv")

# After axial coding you keep a small mapping file: trace_id -> failure_mode.
# (Or add a 'failure_mode' column directly in the review UI once the taxonomy exists.)
modes = pd.read_csv("axial_codes.csv")        # columns: trace_id, failure_mode
df = df.merge(modes, on="trace_id", how="left")

n = len(df)
fails = df[df.verdict == "fail"]
print(f"pass rate: {(df.verdict == 'pass').mean():.1%}  ({n - len(fails)}/{n})")

# Count failure modes (first failure only, so each failing trace counts once)
counts = (fails.failure_mode.value_counts()
          .rename_axis("failure_mode").reset_index(name="traces"))
counts["share_of_failures"] = (counts.traces / len(fails)).round(3)
counts["share_of_all_traces"] = (counts.traces / n).round(3)
print(counts.to_string(index=False))

# Slice by a synthetic dimension or data source: where do failures concentrate?
by_persona = pd.crosstab(df.persona, df.verdict, normalize="index").round(2)
print(by_persona)

# Sanity check: failing traces that never got a failure mode
unassigned = fails[fails.failure_mode.isna()]
assert unassigned.empty, f"{len(unassigned)} failing traces still need an axial code"

Output for the worked example would begin like this (made-up data):

pass rate: 62.5%  (75/120)
             failure_mode  traces  share_of_failures  share_of_all_traces
    refund_outside_policy      12              0.267                0.100
     wrong_order_selected       9              0.200                0.075
 hallucinated_policy_promise    5              0.111                0.042
...

Compute a transition failure matrix

If your review UI (or a per-step code check) records the last successful and first failed state, the matrix is one crosstab. If instead you have per-step pass/fail results for each trace, you can derive both states automatically:

import pandas as pd

STATES = ["Start", "Classify", "Clarify", "Lookup", "Policy",
          "Refund", "Escalate", "Respond"]

# steps.csv: one row per (trace_id, step_index, state, ok) from per-step checks.
steps = pd.read_csv("steps.csv").sort_values(["trace_id", "step_index"])

def first_failure(group):
    """Return (last successful state, first failed state), or None if all ok."""
    last_ok = "Start"
    for state, ok in zip(group.state, group.ok):
        if not ok:
            return pd.Series({"last_ok_state": last_ok, "first_failed_state": state})
        last_ok = state
    return pd.Series({"last_ok_state": None, "first_failed_state": None})

pairs = steps.groupby("trace_id").apply(first_failure).dropna()

matrix = pd.crosstab(
    pd.Categorical(pairs.last_ok_state, categories=STATES[:-1], ordered=True),
    pd.Categorical(pairs.first_failed_state, categories=STATES[1:], ordered=True),
    rownames=["last_ok"], colnames=["first_fail"], dropna=False,
)
print(matrix)

# The hottest cells are where to look first
hot = matrix.stack().sort_values(ascending=False).head(3)
print(hot)
# last_ok  first_fail
# Lookup   Policy        12
# Classify Lookup         7
# Refund   Respond        6

# Optional heatmap:
# import matplotlib.pyplot as plt
# plt.imshow(matrix, cmap="Reds"); plt.xticks(range(len(matrix.columns)), matrix.columns, rotation=45)
# plt.yticks(range(len(matrix.index)), matrix.index); plt.colorbar(); plt.show()

Once the per-step checks are automated, you can run this over thousands of production traces every week and watch whether the hot cells cool down after each fix. That is error analysis turned into a monitoring tool.

Common mistakes

Check yourself

Sources and further reading

• Hamel Husain and Shreya Shankar, Frequently Asked Questions (And Answers) About AI Evals (updated 2026), including the entries on error analysis, how many traces, synthetic data, the benevolent dictator, custom review interfaces and evaluating agentic workflows (transition failure matrices).
• Hamel Husain, Your AI Product Needs Evals.
• Hamel Husain, guide to building LLM-as-a-judge evaluators (critique shadowing, principal domain expert), Oct 29, 2024.
• Shreya Shankar et al., Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences, UIST 2024 (criteria drift, EvalGen).
• Anthropic Engineering, Demystifying evals for AI agents, Jan 9, 2026.
• Anthropic Engineering, Writing effective tools for agents, with agents, Sept 11, 2025.
• Eugene Yan, Evaluating the Effectiveness of LLM-Evaluators, Aug 2024.