A free, visual course

Agent Evals: Zero to Expert

Learn how to measure whether an AI agent actually works, from your first "is this answer right?" check to the dangerous-capability evaluations frontier labs run before a launch. Every modality: tool calls, code, browsers and desktops, voice, images and video.

Start Module 01 →   or open the Eval Playground

Tasks → Agent ⇄ Environment → Graders → Metrics → better Tasks · drag to rotate
28modules, beginner to expert
400+hand-drawn diagrams
12interactive widgets
7agent modalities covered
5runnable example evals
I · Foundations 01 What an eval is 02 Anatomy of an eval 03 Graders 04 Metrics & statistics the four parts of every eval II · Agent evals 05 Why agents differ 06 Tools, RAG, chat 07 Coding agents 08 Web & computer use 09 Voice 10 Vision, image, video 11 Multi-agent, long tasks III · Practice 12 Error analysis 13 LLM-as-a-judge 14 Tools landscape 15 Production & CI build evals for real products IV · Expert 16 How labs do evals 17 Safety evals 18 Benchmark pitfalls 19 Capstone 20 Expert drill beginner → intermediate → advanced → expert V · Building blocks parts inside every agent 21 Harness 22 Skills 23 Memory 24 Tools & MCP VI · Beyond the core security, real work, domains, law 25 Security 26 Research & work 27 Domains 28 Governance
The path. Twenty-eight modules in six parts. Part V zooms in on the building blocks every agent shares: its harness, skills, memory, and tools. Part VI goes beyond the core: agent security, deep research and real work, domain evals, and governance. Each one is built around diagrams, with code you can run and a short self-check at the end.

How to use this course

Visual first

Every concept comes with a diagram of the mechanism: data flow, loops, side-by-side comparisons, real computed charts. Read the picture, then the prose.

Then precise

Definitions, formulas and runnable Python follow the intuition. Code uses real, current library APIs (Inspect, the Anthropic and OpenAI SDKs, Ragas, and more).

Then expert

"Expert corner" callouts cover the debates, failure modes and the way frontier labs and safety institutes actually run evals, with sources.

What you'll be able to do

  • Design an eval for any agent: tool-using assistants, coding agents, browser and desktop agents, voice agents, image and video agents, multi-agent systems.
  • Choose and validate graders, including LLM judges you can actually trust.
  • Evaluate the building blocks inside an agent on their own: the harness around the model, the skills it loads, its long-term memory, and the tools and MCP servers it calls.
  • Test agents against prompt injection and tool poisoning, grade deep-research reports and real work deliverables, and build expert-rubric evals for health, law, finance and data.
  • Produce the kind of eval evidence regulators and standards ask for: EU AI Act duties, NIST AI RMF, ISO/IEC 42001, and safety cases.
  • Report results with honest error bars, and know when a difference is real.
  • Read any benchmark paper or model system card critically: what it measures, how it is scored, how it can be gamed.
  • Pick the right tool (Inspect, promptfoo, Braintrust, LangSmith, Langfuse, DeepEval, Ragas, OpenAI Evals and others) and explain how they differ.
  • Explain how Anthropic, OpenAI, Google DeepMind and independent evaluators like METR and the UK AI Security Institute evaluate frontier models.

Modules

About

Written for a motivated beginner with no background in machine learning evaluation. Facts about tools, benchmarks and lab practices were checked against primary sources (papers, official docs, system cards) in September 2026 and are cited at the end of each module. This field moves fast: when a number matters, follow the link and check the current value.

Content licensed CC BY 4.0. Code snippets MIT.