Graders: code, model, human

The grader is the part of an eval that decides "good or bad". Choose it badly and every number downstream is fiction. This module walks the whole spectrum, from a one-line string comparison to a panel of human experts, and shows how to pick, build and verify the right one.

What a grader is (and why it is the weakest link)

Recall the eval pipeline from Module 01: dataset, system under test, grader, metric. The grader (also called a scorer, judge or evaluator) is a function that takes the system's output (and often the input, a reference answer, or the whole trajectory) and returns a score. The score might be a boolean (pass/fail), a number (0.73), a label ("correct", "partially correct", "refused") or a ranking ("A is better than B").

In code, every grader has the same shape, whatever is inside it:

def grade(sample, output) -> Score:
    # sample: the dataset row (input, reference answer, rubric, starting state...)
    # output: what the system produced (final text, tool calls, end state, transcript)
    # returns: pass/fail, a number, or a label, ideally with an explanation
    ...

The system under test gets most of the attention, but the grader deserves at least as much. Here is why: your metric can only be as accurate as your grader. If the grader wrongly passes 8% of bad outputs and wrongly fails 5% of good ones, then a 3-point "improvement" in your agent could be pure grader noise. And unlike the agent, nobody is watching the grader. It fails silently.

The grader spectrum

Anthropic's guide to agent evals groups graders into three families: code-based, model-based and human. It is useful to split out a fourth family, similarity metrics, because they look like code but behave very differently. Moving left to right along the spectrum, graders get more expensive and slower, but can judge more subtle qualities.

We will walk through each family, then put them together.

Code-based graders

A code-based grader is an ordinary program with deterministic logic. Anthropic's agent-eval guide lists their strengths as "fast, cheap, objective, reproducible, easy to debug" and their main weakness as brittleness: they reject valid answers that are phrased or structured differently from what the author expected. The whole craft of code grading is to make checks strict about what matters and lenient about what does not.

Exact match

The simplest grader: output == reference. It is perfect when the output space is tiny and controlled, for example a classifier that must output one of five labels, a multiple-choice letter, or a tool name. It fails the moment formatting varies. "Paris", "Paris.", "paris" and "The answer is Paris" are four exact-match failures for one correct answer.

Normalized match

Normalization means transforming both the output and the reference into a canonical form before comparing. The classic recipe (used by the SQuAD question-answering benchmark and copied widely) is: lowercase, strip punctuation, remove articles ("a", "an", "the"), and collapse whitespace.

Normalization is a design decision, not a free lunch. Lowercasing is harmless for city names but wrong for case-sensitive answers like SQL identifiers, file paths or chemical formulas ("Co" is cobalt, "CO" is carbon monoxide). Only normalize away differences that genuinely do not matter for your task.

Regex and "contains"

A regular expression (regex) grader checks whether the output matches a pattern: an order ID of the form ORD-\d{6}, a date in ISO format, or a final line that reads ANSWER: X. A "contains" grader checks for a substring. These are useful for checking one fact inside a longer response, and for negative checks ("the reply must not contain the internal system prompt" or "must not mention a competitor").

The danger is over-matching. A grader that passes if the output contains "42" also passes "I'm not sure, maybe 41, 42 or 43". Many benchmarks solve this by asking the model to put its final answer in a fixed format (for example the last line, or inside \boxed{} for maths) and grading only that extracted part.

Numeric tolerance

For numeric answers, compare numbers, not strings. "1,000", "1000.0" and "1e3" are all the same number. Extract the number, then compare with a tolerance: an absolute tolerance (within ±0.01) for small quantities, or a relative tolerance (within 0.1%) for large ones. Python's math.isclose supports both.

import math
import re
import string

def exact_match(output: str, reference: str) -> bool:
    return output == reference

def normalize(text: str) -> str:
    """SQuAD-style normalization: lowercase, drop punctuation and articles, squash spaces."""
    text = text.lower()
    text = "".join(ch for ch in text if ch not in string.punctuation)
    text = re.sub(r"\b(a|an|the)\b", " ", text)
    return " ".join(text.split())

def normalized_match(output: str, reference: str) -> bool:
    return normalize(output) == normalize(reference)

def regex_match(output: str, pattern: str) -> bool:
    return re.search(pattern, output) is not None

NUMBER = re.compile(r"-?\d+(?:,\d{3})*(?:\.\d+)?(?:[eE][-+]?\d+)?")

def extract_last_number(text: str) -> float | None:
    matches = NUMBER.findall(text)
    return float(matches[-1].replace(",", "")) if matches else None

def numeric_match(output: str, reference: float,
                  rel_tol: float = 1e-3, abs_tol: float = 1e-6) -> bool:
    value = extract_last_number(output)          # grade the LAST number: usually the answer
    if value is None:
        return False
    return math.isclose(value, reference, rel_tol=rel_tol, abs_tol=abs_tol)

# Quick checks
assert normalized_match("The Eiffel Tower!\n", "Eiffel Tower")
assert numeric_match("So the total is $1,234.50.", 1234.5)
assert not numeric_match("Somewhere between 1,000 and 2,000", 1234.5)

JSON and schema validation

Agents constantly produce structured output: tool-call arguments, extracted records, API payloads. A schema grader checks that the output parses as JSON and conforms to a JSON Schema (a standard way of declaring "this object must have an order_id string and an amount number ≥ 0"). Schema validity is a necessary condition, not a sufficient one: a perfectly valid object can still hold the wrong values. So schema graders are usually paired with field-level checks against the reference.

import json
from jsonschema import Draft202012Validator   # pip install jsonschema

REFUND_SCHEMA = {
    "type": "object",
    "properties": {
        "order_id": {"type": "string", "pattern": "^ORD-[0-9]{6}$"},
        "amount":   {"type": "number", "minimum": 0},
        "reason":   {"type": "string", "enum": ["damaged", "late", "wrong_item", "other"]},
    },
    "required": ["order_id", "amount", "reason"],
    "additionalProperties": False,
}
validator = Draft202012Validator(REFUND_SCHEMA)

def grade_refund_json(output: str, expected: dict) -> dict:
    # 1. Does it parse at all?
    try:
        data = json.loads(output)
    except json.JSONDecodeError as e:
        return {"passed": False, "stage": "parse", "errors": [str(e)]}
    # 2. Is it the right shape?
    errors = [f"{list(err.path)}: {err.message}" for err in validator.iter_errors(data)]
    if errors:
        return {"passed": False, "stage": "schema", "errors": errors}
    # 3. Are the values right? (the part people forget)
    wrong = [k for k in ("order_id", "reason") if data[k] != expected[k]]
    if abs(data["amount"] - expected["amount"]) > 0.005:
        wrong.append("amount")
    return {"passed": not wrong, "stage": "values", "errors": wrong}

Notice the grader reports which stage failed. That tiny detail pays off during error analysis (Module 12): "30% of failures are schema errors" points to a completely different fix than "30% have the wrong amount".

Execution-based graders: unit tests and state checks

For many agent tasks, the most trustworthy grader does not read the output at all. It runs it, or inspects the world after the agent has acted.

• Unit-test graders execute generated code against tests. HumanEval and MBPP do this for single functions. SWE-bench does it for whole repositories: an agent's patch must make the FAIL_TO_PASS tests pass (the bug is fixed) while keeping the PASS_TO_PASS tests passing (nothing else broke). See Module 07.
• State-check graders compare the environment's final state with a goal state. τ-bench compares the final database with an annotated goal database. AppWorld runs "state-based unit tests" on the database diff, including checks for collateral damage (changes the agent should not have made). The agent can phrase its final message however it likes; what counts is what actually happened.

Execution-based grading is the gold standard for agents because it measures the outcome the user cares about and is immune to confident-sounding prose. It is also the most work to build, since you need a reproducible sandbox, and it can be wrong in its own way: tests can be too weak (bad code passes) or too strict (good code fails).

import subprocess
import sys
import tempfile
from pathlib import Path

def grade_with_unit_tests(candidate_code: str, test_code: str, timeout_s: int = 60) -> dict:
    """Run hidden tests against model-written code. ALWAYS do this inside a sandbox
    (container / VM with no network and no secrets): the code is untrusted."""
    with tempfile.TemporaryDirectory() as tmp:
        Path(tmp, "solution.py").write_text(candidate_code)
        Path(tmp, "test_solution.py").write_text(test_code)
        try:
            proc = subprocess.run(
                [sys.executable, "-m", "pytest", "-q", "test_solution.py"],
                cwd=tmp, capture_output=True, text=True, timeout=timeout_s,
            )
        except subprocess.TimeoutExpired:
            return {"passed": False, "reason": "timeout"}
        return {
            "passed": proc.returncode == 0,          # 0 means every test passed
            "log_tail": proc.stdout[-2000:],         # keep evidence for debugging
        }

def grade_refund_state(db, order_id: str) -> dict:
    """State check: did the agent actually do the thing, and nothing else?"""
    row = db.execute("SELECT status, refund_amount FROM orders WHERE id = ?", (order_id,)).fetchone()
    others = db.execute("SELECT COUNT(*) FROM orders WHERE id != ? AND status = 'refunded'",
                        (order_id,)).fetchone()[0]
    return {
        "refund_issued": row is not None and row[0] == "refunded",
        "no_collateral_damage": others == 0,     # AppWorld-style collateral check
    }

Similarity metrics, and why they are weak for LLMs

Before LLMs, text generation (translation, summarization) was graded with overlap metrics that compare the output to one or more human-written reference texts:

• BLEU (2002, machine translation) counts how many of the output's n-grams (runs of 1, 2, 3, 4 words) appear in the reference: roughly a precision measure, with a penalty for being too short.
• ROUGE (2004, summarization) counts how many of the reference's n-grams appear in the output: roughly a recall measure. ROUGE-L uses the longest common subsequence.
• Embedding similarity turns both texts into vectors with an embedding model and computes the cosine of the angle between them. BERTScore does a token-by-token version. These capture synonyms and paraphrase better than n-grams.

They are cheap and deterministic, but they measure surface resemblance to one particular reference, not correctness. LLM outputs are long, varied and often phrased nothing like the reference. Worse, a tiny word change can flip the meaning while barely moving the score.

Embedding similarity handles paraphrase better (A and the reference would land close together) but is still fooled by negation and by missing details, because "moved to Friday" and "not moved to Friday" are about the same topic and so sit near each other in embedding space. Where similarity metrics still earn their place:

• Tasks with a narrow, well-defined output, such as translation of short strings or near-extractive answers, where a fuzzy match threshold catches typos.
• Cheap monitoring signals: a sudden drop in average similarity to past good answers can flag drift, even if it cannot say what is wrong.
• Retrieval evaluation, where embedding similarity is part of the system anyway.

For grading whether an LLM's answer is correct, helpful or safe, prefer a code check on extracted facts, or an LLM judge.

Model-based graders (LLM-as-a-judge)

A model-based grader uses a language model to read the output and decide. You write a prompt that contains the task, the output and grading instructions, and the judge returns a score or verdict. Anthropic's guide sums up the trade: model graders are "flexible, scalable" and capture nuance, but are "non-deterministic", cost money per call, and need calibration against humans. This section is an introduction; Module 13 goes deep.

Four ways to ask a judge

Zheng et al. (2023), the paper that introduced MT-Bench and studied "LLM-as-a-judge" systematically, used both single-answer grading and pairwise comparison with GPT-4 as the judge. They found GPT-4's judgments agreed with human experts over 80% of the time, about the same rate at which humans agreed with each other. That result is why LLM judges became mainstream. The same paper also documented the biases we cover later.

Anatomy of a good judge prompt

1. Role and task: "You are grading a customer-support reply." Say exactly what is being graded (the final reply? the whole transcript?).
2. The material: the user's request, the output, and any reference answer or policy, each wrapped in clear tags (<reply>…</reply>) so the judge cannot confuse them.
3. Explicit criteria: a short list of binary checks (next section), each with a one-line definition and, ideally, an example of pass and fail.
4. Reasoning before verdict: ask the judge to write its analysis first and the verdict last. A language model generates left to right; if the verdict comes first, the "reasoning" is written to justify a decision already made. Reasoning first lets the verdict depend on the analysis.
5. Structured output: force a machine-readable format (JSON with a fixed verdict enum) so parsing never fails.
6. A tie-break rule: "If you are unsure, FAIL" (or PASS, depending on which error is costlier for you). Without one, judges tend to be lenient.

Code: a PASS/FAIL judge with the Anthropic SDK

This judge uses structured outputs (output_config.format with a JSON schema) so the response is always valid JSON. The reasoning field is declared first so the model writes it before the verdict.

import json
import anthropic   # pip install anthropic

client = anthropic.Anthropic()          # reads ANTHROPIC_API_KEY from the environment
JUDGE_MODEL = "claude-opus-5-5"         # pin the judge model: changing it changes your metric

RUBRIC = """\
R1 Resolves the request: the reply addresses what the customer actually asked.
R2 Policy-correct: any refund/return promise matches the POLICY exactly.
R3 No fabrication: every order detail in the reply appears in ORDER_DATA.
R4 Clear next step: the customer knows what happens next or what to do.
R5 Professional tone: no blame, sarcasm or excessive apology."""

JUDGE_PROMPT = """You are grading one reply written by a customer-support agent.

<policy>{policy}</policy>
<order_data>{order_data}</order_data>
<customer_message>{customer_message}</customer_message>
<agent_reply>{agent_reply}</agent_reply>

Grade the agent_reply against each rubric item below. For each item, quote the
relevant part of the reply and decide whether it is met.
<rubric>
{rubric}
</rubric>

The verdict is PASS only if every rubric item is met. If you are unsure whether
an item is met, treat it as not met. Write your reasoning first, then the verdict."""

VERDICT_SCHEMA = {
    "type": "object",
    "properties": {
        "reasoning":     {"type": "string"},                          # generated first
        "failed_checks": {"type": "array", "items": {"type": "string"}},
        "verdict":       {"type": "string", "enum": ["PASS", "FAIL"]}, # generated last
    },
    "required": ["reasoning", "failed_checks", "verdict"],
    "additionalProperties": False,
}

def judge_reply(policy: str, order_data: str, customer_message: str, agent_reply: str) -> dict:
    response = client.messages.create(
        model=JUDGE_MODEL,
        max_tokens=16000,
        output_config={
            "effort": "medium",
            "format": {"type": "json_schema", "schema": VERDICT_SCHEMA},
        },
        messages=[{
            "role": "user",
            "content": JUDGE_PROMPT.format(
                policy=policy, order_data=order_data, rubric=RUBRIC,
                customer_message=customer_message, agent_reply=agent_reply,
            ),
        }],
    )
    if response.stop_reason == "refusal":          # count grader errors separately,
        return {"verdict": "ERROR", "reasoning": "judge refused", "failed_checks": []}
    text = next(block.text for block in response.content if block.type == "text")
    return json.loads(text)

result = judge_reply(
    policy="Damaged items: full refund within 30 days of delivery. No return needed.",
    order_data="ORD-104233, delivered 2026-09-12, mug, $18.00",
    customer_message="My mug arrived cracked. Can I get my money back?",
    agent_reply="So sorry about that! I've issued a full $18.00 refund to your card; "
                "it should appear in 3-5 business days. No need to send the mug back.",
)
print(result["verdict"], result["failed_checks"])

Things to notice: the judge model is pinned (a silent judge upgrade changes your metric as surely as a dataset change would), refusals are recorded as ERROR rather than silently counted as failures, and the rubric lives in one constant you can version. In a real harness you would also log the full reasoning so you can audit disagreements.

Human graders

Human grading is the reference point that every other grader is ultimately checked against. Anthropic's guide calls it "gold standard quality" and also "expensive, slow". There are three main forms:

Elo and Bradley–Terry both model the probability that A beats B as a function of the difference between their hidden "strength" scores: P(A beats B) = 1 / (1 + 10^{(R_B − R_A)/400}) in Elo's chess notation. Elo updates ratings one game at a time (so the order of games matters); Bradley–Terry fits all ratings at once to all votes, which is more stable. Arena-style preference ranking is powerful for "which model do people like more?", but preference is not correctness: raters can reward confident, long, nicely formatted answers. LMArena added "style control" to adjust for length and markdown for exactly this reason, and critics such as "The Leaderboard Illusion" (Singh et al., 2025) have questioned how private testing affects the rankings. See Module 18.

Choosing a grader

With the families laid out, here is how to choose. The guiding rule, shared by Anthropic's and OpenAI's eval guidance: use the cheapest grader that is reliable for the question you are asking, and escalate only when it cannot see what matters.

Trade-offs at a glance

Cost figures are orders of magnitude to guide intuition, not quotes; they depend on the model, the length of what is graded and your labor costs.

The same ideas in real frameworks

Every eval framework ships ready-made graders from this spectrum. UK AISI's Inspect has scorers such as match(), includes() and model_graded_qa(); promptfoo has assertion types like equals, contains, contains-json, similar and llm-rubric; Braintrust's open-source autoevals library has ExactMatch plus LLM-judge scorers such as Factuality. Module 14 compares them. One well-documented example is worth studying closely because it names each grader type explicitly: OpenAI's Evals API.

Case study: OpenAI's Evals API grader types

An OpenAI eval had a data_source_config (the JSON schema of each dataset item) and a list of testing_criteria, each one a grader. Graders refer to the dataset row with {{ item.field }} and to the model's output with {{ sample.output_text }}.

from openai import OpenAI
client = OpenAI()

# 1. string_check: exact / contains. operation is "eq", "neq", "like" (contains) or "ilike" (case-insensitive contains)
label_exact = {
    "type": "string_check", "name": "category matches label",
    "input": "{{ sample.output_text }}", "operation": "eq",
    "reference": "{{ item.correct_label }}",
}

# 2. text_similarity: overlap metrics. evaluation_metric is one of fuzzy_match, bleu, gleu,
#    meteor, cosine, rouge_1 ... rouge_5, rouge_l
summary_overlap = {
    "type": "text_similarity", "name": "summary ROUGE-L",
    "input": "{{ sample.output_text }}", "reference": "{{ item.reference_summary }}",
    "evaluation_metric": "rouge_l", "pass_threshold": 0.6,
}

# 3. score_model: pointwise numeric judge
quality_score = {
    "type": "score_model", "name": "reference-guided correctness",
    "model": "o4-mini-2025-04-16",
    "input": [
        {"role": "system", "content": "You are an expert grader. Return 1 if the answer "
                                      "states the same facts as the reference, else 0."},
        {"role": "user", "content": "Reference: {{ item.reference_answer }}\n"
                                    "Answer: {{ sample.output_text }}"},
    ],
    "range": [0, 1], "pass_threshold": 0.5,
}

# 4. label_model: categorical judge; some labels count as passing
policy_label = {
    "type": "label_model", "name": "policy compliance",
    "model": "o3-mini",
    "input": [
        {"role": "developer", "content": "Label the reply as compliant or non_compliant "
                                         "with the refund policy."},
        {"role": "user", "content": "Policy: {{ item.policy }}\nReply: {{ sample.output_text }}"},
    ],
    "labels": ["compliant", "non_compliant"],
    "passing_labels": ["compliant"],
}

# 5. python: arbitrary code; grade(sample, item) returns a float
has_order_id = {
    "type": "python", "name": "mentions the order id",
    "source": (
        "def grade(sample, item) -> float:\n"
        "    return 1.0 if item['order_id'] in sample['output_text'] else 0.0\n"
    ),
}

eval_obj = client.evals.create(
    name="Support replies",
    data_source_config={
        "type": "custom",
        "item_schema": {
            "type": "object",
            "properties": {
                "ticket": {"type": "string"}, "correct_label": {"type": "string"},
                "reference_answer": {"type": "string"}, "reference_summary": {"type": "string"},
                "policy": {"type": "string"}, "order_id": {"type": "string"},
            },
            "required": ["ticket", "correct_label"],
        },
        "include_sample_schema": True,
    },
    testing_criteria=[label_exact, summary_overlap, quality_score, policy_label, has_order_id],
)

A sixth type, multi, combined several graders' scores with a formula (calculate_output), which is how you would weight, say, correctness at 0.7 and format at 0.3. The same five ideas map directly onto promptfoo assertions (equals, similar, llm-rubric, javascript/python), so the taxonomy survives the migration even though this API does not.

Outcome vs process grading (a preview)

For agents there is a second choice besides which grader: what it looks at. An outcome grader checks the end result (the refund exists in the database, the tests pass, the file was created). A process or trajectory grader checks the path: which tools were called, in what order, with what arguments, and whether any step broke a rule.

Anthropic's agent-eval guidance leans towards outcomes: checking for "very specific steps like a sequence of tool calls in the right order" is "too rigid and results in overly brittle tests, as agents regularly find valid approaches that eval designers didn't anticipate." The practical synthesis is: grade outcomes by default, and add targeted process checks only for rules that genuinely matter (safety steps, forbidden actions, cost or step limits), phrased as constraints ("no refund before verification") rather than as a required script. The same guide recommends partial credit for multi-part tasks: an agent that verified the customer and found the problem but failed to refund is meaningfully better than one that failed immediately. Module 05 develops trajectory evaluation fully.

Rubric design

A rubric is the written definition of "good" that a grader (human or model) applies. Rubric quality is the single biggest lever on judge quality: a strong judge with a vague rubric is worse than a modest judge with a sharp one.

Binary checks beat 1–10 scales

The instinctive rubric is "rate helpfulness from 1 to 10". Practitioners who have built many judges, such as Hamel Husain, argue strongly against it: "If your evaluations consist of a bunch of metrics that LLMs score on a 1-5 scale (or any other scale), you're doing it wrong." The reasons:

1. Nobody agrees on what the middle numbers mean. Is a 6 "acceptable with flaws" or "mostly bad"? Two humans, or the same judge on two days, place the same output at 6 and 8. Binary questions ("did it state the refund amount?") leave far less room for interpretation, so agreement goes up.
2. Scores cluster. Judges and people both avoid the extremes, so most outputs land at 6–8 and the scale's real resolution is two or three points, not ten.
3. Not actionable. As Husain puts it, "people don't know what to do with a 3 or 4." A failed binary check names the problem.
4. Easier to validate. Checking a binary judge against human labels is a simple confusion matrix (below). Validating a 10-point judge requires correlation statistics and still hides where it goes wrong.
5. Forces a decision about what matters. Writing "PASS means…" makes you decide what "good" is, which is the real work of eval design.

You do not lose granularity: a set of binary checks gives a fraction passed per sample, and pass rates per check across the dataset. If you truly need graded quality (for example a leaderboard of creative writing), use pairwise comparison rather than absolute scores; relative judgments are easier for both humans and models.

Analytic vs holistic rubrics

Worked example: a rubric for a support agent

Here is a complete analytic rubric for the refund-handling support agent used throughout this course. Notice that each check is observable in the transcript, has a definition sharp enough for two graders to agree, and says which grader type should enforce it.

Overall: a sample passes only if R0, R1, R3 and R7 pass (the "must-haves" where failure causes real harm) and at least two of R2, R4, R5, R6 pass. Report per-check pass rates too, because "R5 fails 40% of the time" is a to-do item, whereas "71% overall" is not. Where do checks like these come from? Not from brainstorming, but from reading real failures and naming the patterns you see: that process, error analysis, is the subject of Module 12.

How graders fail

Treat every grader as a measuring instrument with an error rate. There are four ways it goes wrong.

False positives and false negatives

• A false positive (false pass) is a bad output the grader accepts: a regex that finds "42" inside a hedge, tests too weak to catch a bug, a lenient judge. False passes inflate your score and hide real failures; they are the most dangerous kind because nobody investigates a pass.
• A false negative (false fail) is a good output the grader rejects: exact match on "Paris." vs "Paris", tests that demand one particular implementation, a judge that penalizes a valid alternative approach. False fails deflate your score and send engineers chasing non-existent bugs.

Both types also distort comparisons. If a new model phrases answers differently, a brittle grader can make it look worse when it is actually better.

Reward hacking the grader

When a grader is used as a target, and especially when it is used as a reward for reinforcement learning, the system under test learns to satisfy the grader rather than the goal. This is reward hacking, an instance of Goodhart's law ("when a measure becomes a target, it ceases to be a good measure"). Agent examples include editing or deleting the unit tests instead of fixing the code, hard-coding the expected outputs, special-casing the checker, or writing "All tests pass!" in the final message to sway an LLM judge. METR reported in 2025 that recent frontier models sometimes did exactly these things on their tasks. Anthropic's guidance is blunt: "Make your graders resistant to bypasses or hacks. The agent shouldn't be able to easily 'cheat' the eval." Practical defenses:

• Keep grading code, hidden tests and reference answers outside the agent's sandbox, and re-run tests from a clean copy rather than trusting the agent's test run.
• Grade the state, not the agent's claims about the state.
• Add checks for tampering: were test files modified? Did the diff touch the grader?
• Tell the judge to ignore the agent's self-assessment, and read transcripts of suspiciously high-scoring runs.

Judge biases

Zheng et al. (2023) documented three systematic biases in LLM judges, and later work has confirmed and extended them:

Zheng et al. also found judges struggle to grade maths and reasoning they cannot solve themselves, which is why reference-guided grading (giving the judge the correct answer) helps so much there. Other common issues: leniency (a tendency to pass borderline cases), sensitivity to prompt wording, and being persuaded by confident or authoritative language inside the output.

The grader is often the weakest link

Put together, this means your eval's accuracy is capped by its grader's. If you validate your agent carefully (hundreds of samples, repeated trials, confidence intervals from Module 04) but never validate your grader, you have measured noise precisely. Any grader that is not a trivially correct code check needs to be checked against humans before its numbers drive decisions.

Validating a grader against human labels

The method is simple and non-negotiable for LLM judges:

1. Take a sample of real outputs (100–200 is a good start), including plenty of failures. If your system passes 95% of the time, deliberately oversample failures or you will not learn how the grader handles them.
2. Have a domain expert (ideally the person who owns quality for the product) label each one PASS/FAIL using the rubric, without seeing the grader's verdict.
3. Run the grader on the same outputs.
4. Build a confusion matrix and compute agreement statistics. Read every disagreement.
5. Fix the rubric or prompt, and re-measure on a different held-out labeled set so you do not overfit the judge to the examples you tuned on.

What each number tells you:

• Raw agreement (accuracy): the fraction where grader and human match. Easy to read but misleading when classes are imbalanced: if 95% of outputs pass, a grader that always says PASS has 95% agreement and is useless.
• True positive rate (TPR, also recall on passes): of the outputs humans passed, how many did the grader pass? Low TPR means many false fails.
• True negative rate (TNR, also specificity): of the outputs humans failed, how many did the grader catch? Low TNR means failures slip through. Hamel Husain's advice: "treat the human labels as ground truth and report the judge's True Positive Rate and True Negative Rate separately."
• Precision of the grader's verdicts: when the grader says PASS, how often is it right? This matters when you use the grader to filter (for example to pick "good" examples for training).
• Cohen's kappa (κ): agreement corrected for chance. κ = (p_o − p_e) / (1 − p_e), where p_o is observed agreement and p_e the agreement expected if both raters labeled at random with their own pass rates. Here p_e = 0.525 × 0.5 + 0.475 × 0.5 = 0.5, so κ = 0.75. Values above about 0.6 are usually read as substantial agreement, but the interpretation bands are conventions, not laws.

A useful benchmark: compare the judge's agreement with a human to the agreement between two humans on the same items. If humans agree 85% of the time and the judge agrees with them 84% of the time, the judge is about as good as a second human rater, which is the standard Zheng et al. used. You can also use the TPR and TNR to correct an observed pass rate for known grader error, a technique covered with the statistics in Module 13.

Grading multimodal outputs

Agents increasingly speak, see and click. The same grader families apply; what changes is how you get from pixels or waveforms to something gradable.

The key principle carries over: whenever there is a machine-checkable state (a database row, a file, a DOM element), grade that. Use multimodal judges for what is genuinely perceptual (does the voice sound impatient? is the chart legible?), and validate them against humans exactly as you would a text judge. They are newer and generally less reliable.

Putting it together: a grading stack

A realistic grading setup for the support agent in this module might look like this:

1. Gate checks (code, every sample, milliseconds): the run finished without errors; tool arguments pass their schemas; no forbidden tool was called; the reply contains no internal markers (regex).
2. Outcome check (code, every sample): the database state matches the goal state, with no collateral changes.
3. Process rules (code over the trajectory): identity verified before any refund; fewer than 15 tool calls.
4. Quality checks (LLM judge, every sample): R2–R6 from the rubric, binary, reasoning first, structured output, pinned judge model.
5. Calibration (humans, a sample each release): an expert labels 100 fresh samples; you recompute TPR/TNR for each judge check and read every disagreement.

Cheap checks run first and short-circuit: there is no point paying a judge to assess the tone of a reply whose refund amount is wrong. Every check logs its verdict separately so that error analysis can slice by check.

Check yourself

Sources and further reading

• Anthropic, Demystifying evals for AI agents (Jan 2026): code, model and human graders; outcomes over paths; partial credit; resisting hacks; calibrating judges.
• Anthropic, Define success criteria and build evaluations.
• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (NeurIPS 2023 Datasets and Benchmarks): position, verbosity and self-enhancement bias; over 80% agreement with humans.
• Chiang, Zheng et al., Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (2024): pairwise votes and Bradley–Terry ratings.
• Singh et al., The Leaderboard Illusion (2025): critique of arena-style rankings.
• Panickssery, Bowman and Feng, LLM Evaluators Recognize and Favor Their Own Generations (2024): self-preference bias.
• Hamel Husain, Using LLM-as-a-Judge for evaluation: a complete guide: binary pass/fail, critiques, TPR/TNR against an expert.
• OpenAI, Graders guide and Evals guide (both deprecated; shutdown 30 Nov 2026); Evals API regression cookbook (label_model example); migration guide to promptfoo.
• OpenAI, Why SWE-bench Verified no longer measures frontier coding capabilities (Feb 2026): flawed tests as false negatives.
• Jimenez et al., SWE-bench (ICLR 2024): FAIL_TO_PASS and PASS_TO_PASS execution grading.
• Trivedi et al., AppWorld (ACL 2024): state-based unit tests with collateral-damage checks.
• METR, Recent frontier models are reward hacking (2025).
• Papineni et al., BLEU (2002); Lin, ROUGE (2004); Rajpurkar et al., SQuAD (2016; answer normalization).
• UK AI Security Institute, Inspect scorers; promptfoo assertions; Braintrust autoevals.