Eval Playground

Every interactive widget from the course on one page. Drag the sliders until the intuition sticks: most of eval statistics is visible here.

Noise: the same eval gives different scores

Before you celebrate a 3-point gain, see how much a score moves when nothing changed at all. (Module 04)

Error bars

The confidence interval of an accuracy depends mostly on how many samples you have. (Module 04)

Comparing two systems

Paired comparisons on the same questions: the right way to ask "is B better than A?". (Module 04)

pass@k vs pass^k

Capability ("can it ever do this?") vs reliability ("does it do this every time?"). (Modules 04, 05, 06)

Long tasks compound errors

Why long-horizon agents are hard, and why per-step reliability matters. (Module 11)

Judges make mistakes too

Correcting an LLM judge's pass rate using its measured error rates. (Module 13)

Word error rate

The classic speech-recognition metric, with a live alignment. (Module 09)

Arena ratings

How pairwise preference leaderboards turn votes into ratings. (Modules 04, 18)

Ablate an agent harness

Keep the model fixed, switch harness components on and off, and see whether the difference is real. (Module 21)

Skill trigger threshold

Why near-miss negatives matter when you test whether a skill fires. (Module 22)

Memory retrieval trade-off

Top-k retrieval against stuffing the whole history into context. (Module 23)

Infrastructure noise

Same agents, different sandbox resources: when the machine matters more than the model. (Module 18)