What an eval is, and why it matters

An eval is a repeatable experiment that turns "does my AI system work?" into a number you can trust, compare and improve. Everything else in this course is a refinement of that one idea.

The one-sentence definition

An evaluation ("eval") is a fixed set of inputs, run through your AI system, whose outputs are scored by a grader and aggregated into metrics, so that two versions of the system can be compared fairly.

Four words in that definition do the heavy lifting:

• Fixed: if the inputs change between runs, you cannot tell whether the score moved because the system improved or because the test got easier.
• System: you are rarely evaluating "a model". You are evaluating a model plus a prompt, tools, retrieval, memory, a loop that decides when to stop. Anthropic and others call the non-model part the scaffold or harness. Change the scaffold and the score changes, sometimes by more than a model upgrade.
• Grader: something has to decide whether each output is good. It can be a string match, a program that runs unit tests, another LLM, or a human.
• Compared: an eval score is almost meaningless on its own. "71%" tells you little; "71% vs 64% last week, on the same 400 tasks, with a 95% confidence interval of ±4%" tells you a lot.

Evals vs unit tests

If you come from software engineering, the closest thing you know is a test suite. Evals borrow the idea but break three assumptions.

Why "vibe checks" stop working

Almost every AI product starts with a developer typing a few prompts and eyeballing the answers. That is fine on day one. It breaks down for three reasons:

Feel the second problem for yourself. Below, the system never changes; only the luck of the draw does.

OpenAI's own guidance calls this eval-driven development: write the evals first, like tests in test-driven development, then iterate on prompts, tools and models against them. Practitioners such as Hamel Husain put it bluntly: teams that succeed with LLM products are the ones that invest in evals and in looking at their data.

A map of the eval world

"Evals" means different things to different people. Almost all confusion goes away once you ask three questions: who is evaluating, what question are they answering, and at what stage.

Three axes you will see everywhere

Where agents come in

A classic eval is single turn: one prompt in, one answer out, grade the answer. An agent is a model running in a loop: it reads its situation, picks an action (call a tool, click a button, speak, write a file), observes the result, and repeats until it thinks it is done. That changes what you have to evaluate.

This course is about that harder, agentic kind of eval, across every way an agent can act: text and tool calls, code, browsers and desktops, voice, and images and video. We build up the foundations first, because every agent eval is still made of the same four parts: dataset, system, grader, metric.

A short history: from answers to actions

Evals have followed what models can do. When models could only fill in blanks, evals were multiple-choice quizzes. When they could write code, evals ran that code. Now that they act in the world, evals build whole worlds for them to act in.

Notice a pattern that repeats across this course: a benchmark launches with low scores, the field climbs it for a year or two, scores saturate near the ceiling (or flaws surface), and it gets replaced. In February 2026, OpenAI stopped using SWE-bench Verified as a frontier measure after an audit found flawed tests in a majority of the problems it examined. Being an expert means knowing not just the names, but where each eval is in that life cycle.

What makes a good eval

Most evals that fail do so in one of six predictable ways. Use these as a checklist every time you design or read one.

Your first eval, by hand

Before any framework, do one eval with a pencil. Say you're building a weather assistant that must answer with the temperature in Celsius. You write ten test questions, run your assistant once on each, and check every answer yourself.

Result: 8 of 10 pass, 80%. You also learned more than a number: failures cluster into "wrong unit" and "ambiguous city". That is the seed of error analysis (Module 12), the most valuable habit in this whole field.

Now the same eval as code. Nothing magic: a list of cases, a function that calls your system, a grader, an average.

# first_eval.py: the whole idea of an eval in 25 lines
import re

CASES = [
    {"input": "Weather in Paris now?",      "check": "celsius"},
    {"input": "How hot is Phoenix?",        "check": "celsius"},
    {"input": "Weather in Springfield?",    "check": "clarify"},
    # ... more cases
]

def my_agent(question: str) -> str:
    ...  # call your model / agent here and return its reply

def grade(case, reply: str) -> bool:
    if case["check"] == "celsius":          # code-based grader: a regex
        return bool(re.search(r"-?\d+\s*°?\s*C\b", reply))
    if case["check"] == "clarify":          # asks which city?
        return "which" in reply.lower() and "?" in reply
    raise ValueError(case["check"])

results = [grade(c, my_agent(c["input"])) for c in CASES]
print(f"accuracy = {sum(results)}/{len(results)} = {sum(results)/len(results):.0%}")
for c, ok in zip(CASES, results):
    if not ok:
        print("FAIL:", c["input"])            # always look at the failures

Every framework in Module 14 is an industrial version of those 25 lines: better datasets, parallel runs, sandboxed environments, model-based graders, logs you can browse, and statistics on top.

Core vocabulary

Terms are used loosely across the industry. This course uses them like this (closely matching the UK AI Security Institute's Inspect framework and Anthropic's and OpenAI's docs):

Check yourself

Sources and further reading

• OpenAI, Evaluation best practices (eval-driven development).
• Anthropic, Define success criteria and build evaluations.
• UK AI Security Institute, Inspect documentation (task, dataset, solver, scorer vocabulary).
• Hamel Husain, Your AI Product Needs Evals.
• Anthropic, Demystifying evals for AI agents (Jan 2026): task, trial, grader, transcript, outcome; capability vs regression evals.
• OpenAI, Why SWE-bench Verified no longer measures frontier coding (Feb 2026).
• Anthropic, Building effective agents (workflows vs agents).