Evals in production and CI

An offline eval suite tells you whether a change is safe to ship. Production tells you whether you were right. This module wires the two together: tracing, online judges, user signals, CI gates, staged rollouts and dashboards, arranged as one loop that makes your agent a little better every lap.

The big picture: the eval flywheel

So far in this course, an eval has mostly been something you run on your laptop: a fixed dataset, a system, a grader, a number. In a real product, that offline suite is only one station on a loop. The loop looks like this:

1. Offline suite. A curated dataset of tasks, built by error analysis (Module 12), graded by code and calibrated judges (Module 03, Module 13).
2. CI gate. Every change to prompts, tools, model or scaffold runs (part of) the suite automatically. Regressions block the merge.
3. Staged rollout. The change reaches real users gradually: shadow traffic, then a canary, then an A/B test, then everyone.
4. Trace everything. Every production run is recorded as a structured trace: model calls, tool calls, tokens, latency, cost, errors, feedback.
5. Online evals and monitors. Cheap checks run on all traffic; LLM judges score a sample; implicit user signals are aggregated; alerts fire on drops.
6. Review queue. Humans look at the most informative traces: low scores, judge/human disagreements, a random slice.
7. Error analysis. Reviewers cluster failures into a taxonomy and count them.
8. Grow the dataset. Each new failure mode becomes new test cases. The offline suite now covers something it did not cover last week. Back to step 1.

Two properties make the flywheel work. First, the offline suite and the online monitors share graders. If your production judge for "did the agent resolve the refund correctly?" is the same prompt and model as your CI judge, a failure seen in production can be turned into a CI case and graded identically. Second, traces are replayable: a production trace contains enough information (inputs, tool results, configuration) to rebuild the case offline. We will design for both.

Tracing: the raw material

You cannot evaluate what you did not record. Tracing is structured logging for multi-step programs. It comes from distributed-systems observability, where one web request might touch ten services, and it fits agents perfectly because an agent run is also a tree of nested operations.

Spans, traces and sessions

• A span is one timed operation with a name, a start and end time, a status (ok or error), and a bag of key-value attributes. Examples: one model call, one tool execution, one retrieval.
• A trace is a tree of spans that share a trace ID. For an agent, one trace is usually one user turn or one task: the top-level "invoke agent" span, with model calls and tool calls nested inside it.
• A session (also called a conversation or thread) groups the traces of one multi-turn interaction. Many failures only show up at session level: the agent forgot something from turn 2 by turn 6, or the user gave up after three tries.

The OpenTelemetry GenAI semantic conventions

OpenTelemetry (OTel) is the vendor-neutral open standard for traces, metrics and logs. Its semantic conventions are agreed attribute names, so that a span produced by one library can be understood by any backend. The GenAI semantic conventions define names for model calls, agent invocations and tool executions. As of mid-2026 they live in their own repository (open-telemetry/semantic-conventions-genai), and every gen_ai.* attribute is still at Development stability, meaning names can change between releases. Pin the version of the instrumentation library you use, not just "the spec".

Message content (prompts, outputs, tool arguments) is deliberately not captured by default because it routinely contains personal data. Instrumentations expose an opt-in, commonly the environment variable OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT, though implementations vary. This is a real tension: you need content to evaluate quality, and content is where the privacy risk lives. We return to it in the privacy section.

What to log for an agent

Anthropic's guidance on writing tools for agents recommends tracking runtime, number of tool calls, token consumption and tool errors alongside accuracy. In production add configuration and feedback, because you will want to slice every metric by them. A practical checklist:

Code: tracing an agent with Langfuse

Platforms such as Langfuse, LangSmith, Braintrust, Arize Phoenix, MLflow and Weave all give you a tracing SDK plus a UI; most speak OpenTelemetry underneath (Module 14). Here is a tool-using agent loop instrumented with the Langfuse Python SDK. The pattern is identical elsewhere: a root span per turn, a generation span per model call with token usage, a tool span per tool call, and session and user attributes propagated to everything inside.

import anthropic
from langfuse import get_client, observe, propagate_attributes

langfuse = get_client()          # reads LANGFUSE_PUBLIC_KEY / SECRET_KEY / HOST from env
claude = anthropic.Anthropic()
MODEL = "claude-sonnet-5-5"
AGENT_VERSION = "support-bot@2026-09-28"
MAX_STEPS = 20

@observe(name="lookup_order", as_type="tool")      # one tool span per call, args + result captured
def lookup_order(order_id: str) -> dict:
    return orders_api.get(order_id)

@observe(name="issue_refund", as_type="tool")
def issue_refund(order_id: str, amount: float) -> dict:
    return payments_api.refund(order_id, amount)   # a WRITE action: audit these carefully

TOOLS = {"lookup_order": lookup_order, "issue_refund": issue_refund}

def call_model(messages, tool_schemas):
    with langfuse.start_as_current_observation(as_type="generation", name="chat", model=MODEL) as gen:
        resp = claude.messages.create(model=MODEL, max_tokens=1024,
                                      messages=messages, tools=tool_schemas)
        gen.update(output=resp.content,
                   usage_details={"input_tokens": resp.usage.input_tokens,
                                  "output_tokens": resp.usage.output_tokens},
                   metadata={"stop_reason": resp.stop_reason})
        return resp

def handle_turn(user_hash: str, conversation_id: str, messages: list, tool_schemas: list):
    with langfuse.start_as_current_observation(as_type="span", name="support-agent") as root:
        # everything nested below inherits user, session and tags, so you can slice later
        with propagate_attributes(user_id=user_hash, session_id=conversation_id,
                                  tags=["prod", AGENT_VERSION]):
            for step in range(MAX_STEPS):
                resp = call_model(messages, tool_schemas)
                messages.append({"role": "assistant", "content": resp.content})
                if resp.stop_reason != "tool_use":
                    root.update(output=resp.content, metadata={"steps": step + 1, "status": "done"})
                    return resp
                results = []
                for block in resp.content:
                    if block.type == "tool_use":
                        try:
                            out = TOOLS[block.name](**block.input)
                            results.append({"type": "tool_result", "tool_use_id": block.id,
                                            "content": str(out)})
                        except Exception as e:          # tool errors are data, not crashes
                            results.append({"type": "tool_result", "tool_use_id": block.id,
                                            "content": f"error: {type(e).__name__}", "is_error": True})
                messages.append({"role": "user", "content": results})
            root.update(metadata={"steps": MAX_STEPS, "status": "step_cap_hit"})   # alert on this!
            return None

The same structure written directly against OpenTelemetry, using the GenAI attribute names, looks like this. In practice you would usually install an auto-instrumentation package for your model SDK instead of setting attributes by hand; the snippet shows what those packages produce.

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("support-agent")

def run_agent(conversation_id, messages):
    with tracer.start_as_current_span("invoke_agent support_bot") as agent_span:
        agent_span.set_attribute("gen_ai.operation.name", "invoke_agent")
        agent_span.set_attribute("gen_ai.agent.name", "support_bot")
        agent_span.set_attribute("gen_ai.agent.version", AGENT_VERSION)
        agent_span.set_attribute("gen_ai.conversation.id", conversation_id)

        with tracer.start_as_current_span(f"chat {MODEL}") as span:
            span.set_attribute("gen_ai.operation.name", "chat")
            span.set_attribute("gen_ai.provider.name", "anthropic")
            span.set_attribute("gen_ai.request.model", MODEL)
            resp = claude.messages.create(model=MODEL, max_tokens=1024, messages=messages, tools=SCHEMAS)
            span.set_attribute("gen_ai.response.model", resp.model)
            span.set_attribute("gen_ai.usage.input_tokens", resp.usage.input_tokens)
            span.set_attribute("gen_ai.usage.output_tokens", resp.usage.output_tokens)
            span.set_attribute("gen_ai.response.finish_reasons", [resp.stop_reason])

        for block in (b for b in resp.content if b.type == "tool_use"):
            with tracer.start_as_current_span(f"execute_tool {block.name}") as tool_span:
                tool_span.set_attribute("gen_ai.operation.name", "execute_tool")
                tool_span.set_attribute("gen_ai.tool.name", block.name)
                tool_span.set_attribute("gen_ai.tool.call.id", block.id)
                try:
                    TOOLS[block.name](**block.input)
                except Exception as e:
                    tool_span.record_exception(e)
                    tool_span.set_attribute("error.type", type(e).__name__)
                    tool_span.set_status(Status(StatusCode.ERROR))

Online evaluation

Online evaluation means scoring live production traffic, as opposed to a fixed offline dataset. It has three ingredients: deterministic checks on everything, model-based judges on a sample, and user signals wherever users give them. Before the details, one distinction that confuses many teams.

Guardrails vs evals

A guardrail is a check that runs inline, before the user sees the output or before a tool executes, and can block or change what happens. An online eval runs asynchronously, after the fact, and only measures. They can use similar checks (a PII detector can be either), but their engineering constraints are opposite.

Guardrails still need evaluating: measure their precision and recall offline on labeled examples, just like any grader, and monitor their trigger rate online. A guardrail whose trigger rate doubles overnight is telling you something changed, either in the traffic or in the model upstream of it. Eugene Yan notes that fine-tuned small evaluators are a good fit for low-latency guardrails, while bigger judges suit async scoring.

Sampling: what to judge

Running an LLM judge on every production trace is usually wasteful. A good sampling design mixes three streams with different purposes:

1. Deterministic checks on 100%. Free or nearly free: did any tool error? Did the run hit the step cap? Is the output valid JSON? Did the cost exceed a threshold? Did a PII regex fire? These catch the loud failures.
2. A uniform random sample (say 1 to 5%). This is your unbiased estimate of quality over the true traffic distribution. It is the only stream whose average you can put on a dashboard as "the pass rate".
3. Targeted (flagged) samples. Always judge traces with a thumbs-down, an escalation, an error, an unusually long trajectory, a new feature flag, or a slice you are worried about. This stream is where you find failures fastest, but it is biased by design: never average it together with the random stream.

Sizing the random stream (worked example)

How big does the random sample need to be? Use the standard error of a proportion from Module 04. Suppose your agent's true pass rate is about 80% and you judge 4,000 random traces a day:

SE = sqrt(0.8 × 0.2 / 4000) ≈ 0.0063  so the 95% interval is about ± 1.96 × 0.63 ≈ ± 1.2 points.

That is precise enough to see a 3-point daily drop. If you only care about weekly trends, 600 a day (4,200 a week) gives the same precision per week. If you need per-slice numbers (say 10 customer segments), each slice needs its own few hundred to thousand samples, which is where stratified sampling helps: sample small slices at a higher rate, then reweight each slice by its true traffic share when computing the overall number.

Judge cost (worked example)

Say you have 200,000 traces a day, and a judge call reads about 4,000 input tokens (the transcript plus rubric) and writes 300 output tokens. At an illustrative price of $1 per million input tokens and $5 per million output tokens (check your provider's current price sheet):

A twenty-fold saving for almost the same information. Other levers: truncate or summarize long transcripts before judging (but check the judge still agrees with humans on the shortened input), use a cheaper model for a first-pass triage and a stronger one only on uncertain cases, and use provider-side prompt caching for the fixed rubric part of the judge prompt.

Code: a budget-capped online judge sampler

This worker pulls finished traces, decides whether to judge each one, enforces a rate limit and a daily dollar budget, calls a judge, and writes the score back to the trace in Langfuse. Note the deterministic hash sampling: the same trace ID always gets the same decision, so reruns and backfills are reproducible, and the stream label is stored with every score so the dashboard never mixes biased and unbiased samples.

import hashlib, json, time, datetime as dt
import anthropic
from langfuse import get_client

JUDGE_MODEL = "claude-haiku-4-5"       # pick a cheap judge you have CALIBRATED against humans (Module 13)
PRICE_IN, PRICE_OUT = 1.00, 5.00       # USD per million tokens; set from your provider's price sheet
RANDOM_RATE = 0.02                     # 2% uniform sample: the unbiased stream
DAILY_BUDGET_USD = 60.0
MAX_CALLS_PER_MIN = 60

claude = anthropic.Anthropic()
langfuse = get_client()

RUBRIC = """You are grading a customer-support agent transcript.
PASS only if the agent (1) resolved the user's actual request, (2) took no
unauthorised write action, and (3) stated nothing contradicted by tool results.
Think step by step, then answer with JSON: {"pass": true|false, "reason": "..."}"""

class TokenBucket:
    """Allow at most `rate` calls per minute, smoothing bursts."""
    def __init__(self, rate_per_min):
        self.capacity = rate_per_min
        self.tokens = rate_per_min
        self.refill = rate_per_min / 60.0
        self.last = time.monotonic()
    def acquire(self):
        while True:
            now = time.monotonic()
            self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.refill)
            self.last = now
            if self.tokens >= 1:
                self.tokens -= 1
                return
            time.sleep((1 - self.tokens) / self.refill)

class DailyBudget:
    def __init__(self, usd):
        self.usd, self.spent, self.day = usd, 0.0, dt.date.today()
    def can_spend(self, estimate):
        if dt.date.today() != self.day:            # reset at midnight
            self.spent, self.day = 0.0, dt.date.today()
        return self.spent + estimate <= self.usd
    def charge(self, usage):
        self.spent += usage.input_tokens / 1e6 * PRICE_IN + usage.output_tokens / 1e6 * PRICE_OUT

def in_random_sample(trace_id: str, rate: float) -> bool:
    # deterministic: the same trace always gets the same decision
    h = int(hashlib.sha256(trace_id.encode()).hexdigest()[:8], 16) / 0xFFFFFFFF
    return h < rate

def choose_stream(t: dict):
    """Return 'random', 'flagged' or None. Random wins so the unbiased stream stays unbiased."""
    if in_random_sample(t["id"], RANDOM_RATE):
        return "random"
    if t["error"] or t["feedback"] == "down" or t["escalated"] or t["steps"] > 15:
        return "flagged"
    return None

bucket, budget = TokenBucket(MAX_CALLS_PER_MIN), DailyBudget(DAILY_BUDGET_USD)

def judge_trace(t: dict):
    stream = choose_stream(t)
    if stream is None:
        return
    transcript = t["transcript"][-24_000:]                  # cap the input: cost and context control
    est = (len(transcript) / 4 + 600) / 1e6 * PRICE_IN + 400 / 1e6 * PRICE_OUT
    if not budget.can_spend(est):
        if stream == "flagged":
            enqueue_for_human_review(t["id"], reason="judge budget exhausted")
        return                                              # degrade gracefully, never crash
    bucket.acquire()
    resp = claude.messages.create(
        model=JUDGE_MODEL, max_tokens=400, temperature=0,
        system=RUBRIC,
        messages=[{"role": "user", "content": f"<transcript>\n{transcript}\n</transcript>"}],
    )
    budget.charge(resp.usage)
    text = resp.content[0].text
    try:
        verdict = json.loads(text[text.index("{"): text.rindex("}") + 1])
    except ValueError:
        verdict = {"pass": None, "reason": "unparseable judge output"}   # count these; alert if they rise
    if verdict["pass"] is not None:
        langfuse.create_score(
            name=f"judge_resolved_{stream}",           # keep streams separate on dashboards
            value=1.0 if verdict["pass"] else 0.0,
            trace_id=t["id"], data_type="NUMERIC",
            comment=f"{JUDGE_MODEL}: {verdict['reason'][:500]}",
        )
    if verdict["pass"] is not True:
        enqueue_for_human_review(t["id"], reason=f"judge fail ({stream})")

Implicit feedback signals

Explicit feedback (thumbs up or down, star ratings) is precious but rare: typically only a small fraction of users ever click, and those who do are not representative (angry users click more). Implicit signals are behaviors users show anyway, which correlate with success or failure. They are free, available on 100% of traffic, and noisy.

Shipping safely: shadow, canary, A/B, interleaving

Offline evals cannot see everything, so you release changes in stages, each exposing more users and measuring more directly.

Shadow deployments

In a shadow (or "dark launch") deployment, the new version receives a copy of real production requests, but its output is thrown away; users only see the old version. You then compare the two on the same inputs, which is a paired comparison with real traffic, the best of both worlds. The catch for agents: agents act. A shadow agent must not send the email or issue the refund. You have to stub every write tool (return a plausible fake success) and point read tools at real or replayed data. That also means shadow runs diverge from reality after the first write: if the real agent refunded and the shadow did not, turn 2 is no longer comparable. Shadow mode is most informative for single-turn behavior and early trajectory steps.

Canaries

A canary release sends a small fraction of real users (often 1% then 5 to 10%) to the new version, with automated checks that roll back if error rate, cost per trace, p95 latency, step-cap hits or safety-guardrail triggers exceed limits relative to the control. Canaries are for catching breakage quickly, not for measuring small quality differences: 1% of traffic rarely gives enough samples for a precise quality comparison in hours.

A/B tests for agents

An A/B test randomly assigns users to control (A) or treatment (B) and compares outcome metrics. It is the only way to measure causal impact on what the business actually cares about (resolution, retention, revenue). Agent-specific design choices:

• Randomize by user, not by request. Agents have memory and multi-turn sessions; switching versions mid-conversation contaminates both arms. Hash the user ID into an arm and keep it sticky.
• Pick one primary metric (for example "session judged resolved" or "no human escalation within 7 days") and several guardrail metrics that must not degrade (cost per session, p95 latency, safety flags).
• Size it up front. Agents often change behavior by small amounts, and small effects need many users.

Worked example: how many users?

Your baseline resolution rate is 60%. You want 80% power to detect a 2-point absolute improvement at a two-sided 5% significance level. The usual approximation for two proportions:

n per arm ≈ (zα/2 + zβ)² × 2p(1−p) / d² = (1.96 + 0.84)² × 2 × 0.6 × 0.4 / 0.02² ≈ 9,400

So about 9,400 users per arm, around 19,000 in total, and more if users have several sessions (sessions from the same user are correlated, so cluster your standard errors by user, as in Module 04). If you only have 2,000 users a week, a 2-point A/B test takes ten weeks. That is why most day-to-day decisions are made offline, with A/B tests reserved for big changes.

Interleaving and side-by-side

Interleaving is a trick from search and recommendation: instead of showing user group A ranker A and group B ranker B, you show every user a merged list with results from both rankers and see which ranker's items get clicked. Because each user compares both, it needs far fewer users than an A/B test. For agents that produce one answer, pure interleaving rarely applies, but two related ideas do:

• Interleave components. If your agent has a retrieval step or a tool that ranks options, you can interleave those rankings.
• Side-by-side preference. Occasionally show two responses and ask the user which is better. This is a pairwise human preference signal (the same statistics as LMArena, Module 04), but it costs user attention, so use it sparingly and only where both responses are safe to show.

Evals in CI: the gate

Continuous integration (CI) runs automated checks on every code change, typically when a pull request (PR) is opened. Adding evals to CI turns "please remember to run the eval" into "the merge button is disabled until the eval passes". It is where the flywheel's ratchet effect (Module 01) is enforced.

Tiered suites

Agent evals are slow and cost money, so you cannot run everything on every PR. Split into tiers, each with a different trigger, size and job:

This mirrors the capability vs regression distinction from Anthropic's agent-eval guidance: regression evals should stay near 100% and a drop is a signal; capability evals start low and are about headroom. When a capability task becomes reliably solved, "graduate" it into the regression suite.

Thresholds vs statistical regression tests

The simplest gate is an absolute threshold: "pass rate must be at least 90%". It is easy to explain and good for must-pass cases (safety, critical flows), where a single failure really is a blocker. For aggregate quality it has two problems. It ignores noise: with 100 cases, a system with a true pass rate of 91% will score below 90% fairly often just by chance, so the gate flaps. And it ignores change: a drop from 97% to 91% passes the gate even though something clearly broke.

The better gate for aggregates is a statistical regression test: compare the candidate with the baseline on the same cases and ask whether the candidate is credibly worse by more than a tolerance. Because both versions run on the same cases, use a paired method (Module 04): per-case differences cancel out the "some cases are just hard" variance and make the test far more sensitive than comparing two independent averages.

The rule used here: compute the 95% confidence interval of the mean per-case difference (candidate minus baseline) with a paired bootstrap, and fail the build if the lower bound is below −1 point. This is a non-inferiority test: it passes when we are confident the candidate is not more than 1 point worse.

Code: a paired-bootstrap gate over Inspect logs

This script reads an Inspect eval log for the baseline and one for the candidate, averages each case over its epochs, aligns cases by ID, bootstraps the mean difference, prints a summary to the GitHub job page, and exits non-zero if the lower bound is below the tolerance. It also reports per-slice differences using a slice field in each sample's metadata.

#!/usr/bin/env python3
"""evals/compare.py: fail CI if the candidate is credibly worse than the baseline."""
import argparse, os, sys, glob
from collections import defaultdict
import numpy as np
from inspect_ai.log import read_eval_log

def to_float(v):
    # Inspect scorers return numbers or letters: C = correct, I = incorrect, P = partial
    return {"C": 1.0, "I": 0.0, "P": 0.5}.get(v, v) if isinstance(v, str) else float(v)

def load(path, scorer):
    if os.path.isdir(path):                                   # newest log in a directory
        path = max(glob.glob(os.path.join(path, "*.json")), key=os.path.getmtime)
    log = read_eval_log(path)
    assert log.status == "success", f"{path}: eval status {log.status}"
    per_case, slices = defaultdict(list), {}
    for s in log.samples:
        if s.error:                                           # infra errors are not model failures
            continue
        per_case[s.id].append(to_float(s.scores[scorer].value))
        slices[s.id] = (s.metadata or {}).get("slice", "all")
    return {k: float(np.mean(v)) for k, v in per_case.items()}, slices

def paired_bootstrap(diffs, n_boot=10_000, seed=0):
    rng = np.random.default_rng(seed)                         # fixed seed: reproducible CI
    idx = rng.integers(0, len(diffs), size=(n_boot, len(diffs)))
    boots = diffs[idx].mean(axis=1)                           # resample CASES, keep pairs intact
    return diffs.mean(), *np.percentile(boots, [2.5, 97.5])

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--baseline", required=True)
    ap.add_argument("--candidate", required=True)
    ap.add_argument("--scorer", default="includes")
    ap.add_argument("--max-drop", type=float, default=0.01)   # 1 point on a 0-1 scale
    ap.add_argument("--min-coverage", type=float, default=0.95)
    a = ap.parse_args()

    base, _ = load(a.baseline, a.scorer)
    cand, slices = load(a.candidate, a.scorer)
    ids = sorted(set(base) & set(cand))
    coverage = len(ids) / max(len(base), 1)
    if coverage < a.min_coverage:                             # silently missing cases = a broken eval
        sys.exit(f"Only {coverage:.0%} of baseline cases present in candidate run")

    d = np.array([cand[i] - base[i] for i in ids])
    mean, lo, hi = paired_bootstrap(d)
    lines = [f"### Eval gate: {len(ids)} paired cases",
             f"Baseline {np.mean([base[i] for i in ids]):.1%} | Candidate {np.mean([cand[i] for i in ids]):.1%}",
             f"Mean diff **{mean*100:+.1f} pts**, 95% CI [{lo*100:+.1f}, {hi*100:+.1f}]",
             "", "| slice | n | diff (pts) |", "|---|---|---|"]
    by_slice = defaultdict(list)
    for i, x in zip(ids, d):
        by_slice[slices[i]].append(x)
    for s, xs in sorted(by_slice.items()):
        lines.append(f"| {s} | {len(xs)} | {np.mean(xs)*100:+.1f} |")
    passed = lo >= -a.max_drop
    lines.append("\n**PASS**" if passed else
                 f"\n**FAIL**: lower bound {lo*100:+.1f} < {-a.max_drop*100:.1f} pts")
    report = "\n".join(lines)
    print(report)
    if os.environ.get("GITHUB_STEP_SUMMARY"):
        with open(os.environ["GITHUB_STEP_SUMMARY"], "a") as f:
            f.write(report + "\n")
    sys.exit(0 if passed else 1)

if __name__ == "__main__":
    main()

Why bootstrap cases rather than individual epochs? Epochs of the same case are correlated (a hard case is hard every time), so treating them as independent would make the interval too narrow. Averaging within a case first and resampling cases is the simple, correct version of the "cluster your standard errors" advice from Evan Miller's error-bars paper.

Code: the GitHub Actions workflow

The workflow below runs three tiers from one file. On a PR it runs the Inspect smoke suite, compares against the committed baseline with the script above, and also runs a promptfoo prompt-regression matrix with a pass-rate threshold. Nightly it runs the full suite. Weekly it runs the capability suite. PROMPTFOO_PASS_RATE_THRESHOLD sets the minimum pass rate as a percentage (it defaults to 100%); when the rate falls below it, promptfoo eval exits with code 100, which fails the job.

name: agent-evals

on:
  pull_request:
    paths: ["agent/**", "prompts/**", "tools/**", "evals/**"]
  schedule:
    - cron: "0 3 * * *"        # nightly full regression suite
    - cron: "0 4 * * 6"        # weekly capability suite (Saturday)
  workflow_dispatch:

concurrency:                    # a new push cancels the old run: saves money
  group: evals-${{ github.ref }}
  cancel-in-progress: true

jobs:
  smoke:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    timeout-minutes: 20         # hard wall-clock cap
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r evals/requirements.txt
      - name: Run smoke suite on the candidate
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          inspect eval evals/smoke.py --model anthropic/claude-sonnet-5-5 \
            --epochs 3 --max-connections 8 --message-limit 40 \
            --log-format json --log-dir logs/candidate
      - name: Paired-bootstrap gate vs baseline (fail if lower CI bound < -1 pt)
        run: |
          python evals/compare.py --baseline evals/baselines/smoke.json \
            --candidate logs/candidate --scorer includes --max-drop 0.01
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: smoke-logs, path: logs/ }

  prompt-regressions:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: "22" }
      - uses: actions/cache@v4
        with:
          path: ~/.cache/promptfoo
          key: ${{ runner.os }}-promptfoo-v1
      - name: promptfoo matrix, 95% pass-rate gate
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          PROMPTFOO_PASS_RATE_THRESHOLD: "95"
        run: npx promptfoo@latest eval -c evals/promptfooconfig.yaml -o results.json

  nightly-full:
    if: github.event_name == 'schedule' && github.event.schedule == '0 3 * * *'
    runs-on: ubuntu-latest
    timeout-minutes: 120
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r evals/requirements.txt
      - env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          inspect eval evals/full.py --model anthropic/claude-sonnet-5-5 \
            --epochs 5 --log-format json --log-dir logs/nightly
          python evals/compare.py --baseline evals/baselines/full.json \
            --candidate logs/nightly --max-drop 0.01   # non-zero exit = alert on-call
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: nightly-logs, path: logs/ }

  weekly-capability:
    if: github.event_name == 'schedule' && github.event.schedule == '0 4 * * 6'
    runs-on: ubuntu-latest
    timeout-minutes: 360
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r evals/requirements.txt
      - env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: inspect eval evals/capability.py --model anthropic/claude-sonnet-5-5 --log-dir logs/cap
      # no gate: results are uploaded and charted as a trend

A few design notes. The baseline is a committed log that the team refreshes deliberately (for example by a bot PR after a green nightly run on main). That keeps the baseline reviewable, but it means the baseline was produced at a different time from the candidate; if the provider changed the model in between, the difference includes that drift. The more rigorous option is to run baseline and candidate in the same job, which doubles cost but controls for time-varying factors. Many teams do the cheap version per PR and the rigorous version nightly. --message-limit caps each sample's conversation length so a looping agent cannot burn the CI budget (Inspect also supports token and time limits per sample).

Flaky evals

A flaky eval is one that passes and fails on identical code. In ordinary software, a flaky test is a bug. In agent evals, some flakiness is inherent (sampling), so the goal is to measure and contain it rather than eliminate it. Sources and fixes:

Caching and cost budgets

• Response caching. Cache model responses keyed on (model, full prompt, parameters). promptfoo caches by default (the ~/.cache/promptfoo directory in the workflow). Caching is safe for components that did not change, such as the judge scoring an unchanged output. It is dangerous for the system under test: a cached response hides sampling variance, and if the cache key misses something (a tool schema, a system prompt file) you evaluate a stale answer. Rule of thumb: cache judges and fixed fixtures, never cache the candidate agent's own calls in the gating run.
• Provider prompt caching. Agent evals re-send long, identical prefixes (system prompt, tool definitions). Provider-side prompt caching can cut input cost substantially on these runs.
• Budgets at three levels. Per sample (message, token and time limits so one looping case cannot eat the run), per run (a job timeout and a maximum case count), and per month (track spend by suite; a smoke suite that costs $40 per PR will quietly become the largest line item once there are twenty PRs a day).
• Skip what did not change. Use path filters so a README edit does not trigger evals, and consider running only the slices whose tools or prompts were touched, with the full suite nightly as the safety net.

Model and provider migrations

Few changes are as tempting or as risky as swapping the model: a new release is cheaper, faster or smarter on public benchmarks. Public benchmarks do not measure your task (Module 01), and agents are sensitive to behavioral details that benchmarks ignore: how verbose the model is, how eagerly it calls tools, how it formats arguments, when it asks clarifying questions. A migration playbook:

1. Freeze everything else. Same prompts, tools, dataset and graders. Change one variable.
2. Run the full suite with epochs on both models, in the same job, and compare with the paired method. Also run the capability suite: a new model may unlock tasks you had given up on.
3. Compare per slice, not just overall. A flat average can hide a big gain on easy slices and a big loss on a critical one.
4. Compare behavior, not just scores. Tool calls per task, tokens per task, cost per success, latency, refusal rate, clarifying-question rate, finish reasons, output length. These "shape" metrics catch shifts that the pass/fail grader misses and often explain score changes.
5. Read transcripts where the two models disagree. This is where you find out that the new model now asks for confirmation before every refund, which your grader counts as success but your users will hate.
6. Re-tune, then re-evaluate. Prompts are often overfitted to the old model. Allow a round of prompt adjustment, but evaluate the tuned version on a held-out split so you are not overfitting to the eval.
7. Re-validate judges. If the new model is also your judge, or the judge is from the same family, check judge-human agreement again: self-preference bias changes with the pairing (Module 13).
8. Roll out in stages with the new model as the treatment arm, and keep the old one available as a fallback until online metrics confirm.

Worked example: a migration report

The overall number says "ship it". The slices say "the new model is better at explaining policy but worse at refunds". Transcript reading shows it calls lookup_order again before refunding and sometimes hits a rate limit. That is fixable (cache lookups within a session), and fixing it is a better outcome than either blindly shipping or rejecting the model. Illustrative numbers.

Drift: when nothing changed but everything changed

Drift means the world your agent operates in changes after you evaluated it. Your code did not change, but your quality did. Three sources matter for agents:

A fourth, subtler kind is criteria drift: what "good" means changes as you look at more outputs. Shankar and colleagues found that people need criteria to grade outputs, but grading outputs changes their criteria. That is healthy, but it means your rubrics and judges need versioning just like your prompts, and a score series should note when the rubric changed.

Dashboards: what to chart

A dashboard is where the flywheel becomes visible to the whole team. The most common failure is a wall of numbers nobody acts on. Chart a few things, each with a clear owner and a clear action when it moves.

Humans in the loop: review queues and annotation

Every stream above eventually ends in a person reading traces. A review queue (tools such as LangSmith and Langfuse call them annotation queues) is a to-do list of traces with a labeling interface. Done well, it is the highest-value hour of the team's week; done badly, it is a backlog nobody opens. Hamel Husain's advice applies directly: remove all friction from looking at data.

• What goes in: judge failures, cases where the judge was uncertain or unparseable, disagreements between judge and user signal (thumbs-up but judge fail, or the reverse), a small uniform random slice (to find failures no filter catches), and traces from new features.
• What reviewers produce: a binary pass/fail with a short written critique (not a 1-to-5 score), a failure-mode tag from the taxonomy, and a flag "add to dataset". This is exactly the open-coding and axial-coding workflow from Module 12.
• Where labels go: three places. (1) New dataset cases, with the trace converted into a replayable input and the reviewer's critique turned into the expected behavior. (2) Judge calibration data: every human label is also a test of the judge (TPR and TNR, Module 13). (3) Product bugs, when the failure is not the model's fault at all (a broken tool, missing data).
• Quality control: have two reviewers label an overlapping 10% and track agreement (Cohen's kappa). Low agreement means the rubric is ambiguous, not that reviewers are careless.

Privacy and PII in traces

Traces are the most sensitive data many AI teams hold: full conversations, tool results containing customer records, sometimes credentials pasted by users. The same traces are what you want to put in datasets, send to third-party judge models and show to reviewers. Treat this as a design problem from day one:

Agent-specific operations: loops, caps and cost explosions

A chatbot that misbehaves writes one bad paragraph. An agent that misbehaves can call a tool 400 times, spend $80 of tokens on one request, or repeat a write action. Production agents need hard limits that are enforced in code (not requested in the prompt) and monitored like any other SLO.

These controls are themselves things to evaluate. Put a "loop bait" task in the smoke suite (a tool that always returns a retryable error) and assert that the agent stops within the cap and explains itself; put a "large tool output" task in and assert cost stays within budget. Anthropic's tool-writing guidance recommends exactly this kind of tracking of tool calls, tokens and errors alongside accuracy, and notes that what agents omit in transcripts can matter as much as what they include.

Putting it together: a minimum viable production eval stack

If you are starting from nothing, do these in order. Each step is useful on its own.

1. Week 1: tracing with sessions, tool spans, tokens, cost and agent version. Hard step and cost caps in code. Read 50 traces yourself.
2. Week 2: error analysis on those traces; a first offline suite of 30 to 50 cases from real failures; a smoke job in CI with must-pass cases.
3. Week 3: deterministic checks on 100% of traffic; implicit-signal capture; a dashboard with slice success, cost per success, p95 latency, step-cap rate.
4. Week 4: a calibrated LLM judge on a 2% random sample plus flagged traces, with a budget; a weekly review queue that feeds the dataset.
5. Later: paired statistical gate, nightly full suite, canary automation, A/B tests for big changes, anchor-set monitoring of the judge, a capability suite.

Running an eval program

Everything above is machinery: traces, judges, gates, dashboards. Eval programs rarely fail because the machinery is missing. They fail for organizational reasons. Nobody owns the suite, so it goes stale. Nobody reads the dashboard, so drift goes unnoticed. The judge was calibrated once, two models ago. Finance is surprised by the bill, and evals get cut. This section is about running evals as an ongoing program rather than a one-off project.

Who owns evals

The most common failure is an eval suite that is one engineer's side project. When that engineer moves team, the suite quietly dies. Spread ownership across clear roles:

Two organizational shapes work. A central eval team builds shared infrastructure (harness, judges, dashboards) and the evaluation methods. Embedded owners in each product team write the cases and rubrics, because only they know the product. Most mature organizations are a hybrid: platform central, content local. A simple rule keeps either shape honest: every eval has a named owner, and every metric on a dashboard has an action attached ("if this drops below X, Y investigates within a day").

Eval maturity levels

It helps to know where you are. The ladder below is a practical synthesis, not an industry standard. Each level makes the next one possible; skipping levels (for example buying a production monitoring platform before you have read fifty traces yourself) usually produces dashboards nobody trusts.

A quick self-assessment: you are at level 2 if a pull request can fail because of an eval; at level 3 if last month's suite contains cases that came from last month's production failures; at level 4 if you can state your headline metric with a confidence interval and know your judge's agreement with humans; at level 5 if a new feature's success criteria are written as eval cases before the feature is built, and a product decision was changed because of an eval result.

Budgeting eval cost

Evals cost money in four places: per-PR runs, scheduled runs (nightly or weekly full suites), online judging of production samples, and human time for labeling and review. Sandbox compute for agent evals is a fifth line if you run containers at scale. Write the budget down; it turns "evals are expensive" into a list of levers.

"""Monthly eval budget. All prices and volumes are illustrative; plug in your own."""
AGENT_RUN = 0.12      # $ per agent run on one case (tokens, mostly cached prefix)
JUDGE_CALL = 0.01     # $ per judge call
PER_CASE = AGENT_RUN + JUDGE_CALL

def monthly_budget(prs_per_day=15, work_days=22,
                   smoke_cases=40, smoke_epochs=3,
                   nightly_cases=400, nightly_epochs=3,
                   traces_per_day=50_000, judge_sample=0.02,
                   review_hours_per_week=4, hourly_rate=80):
    lines = {
        "smoke gate (per PR)": prs_per_day * work_days * smoke_cases * smoke_epochs * PER_CASE,
        "nightly full suite":  30 * nightly_cases * nightly_epochs * PER_CASE,
        "online judge":        30 * traces_per_day * judge_sample * JUDGE_CALL,
        "human review":        review_hours_per_week * 52 / 12 * hourly_rate,
    }
    lines["total"] = sum(lines.values())
    return lines

for name, dollars in monthly_budget().items():
    print(f"{name:22s} ${dollars:9,.0f}")
# smoke gate (per PR)    $    5,148
# nightly full suite     $    4,680
# online judge           $      300
# human review           $    1,387
# total                  $   11,515

The illustrative result is typical in shape: the per-PR gate is the biggest line, because it multiplies by the number of pull requests, and online judging on a small sample is cheap. The levers follow directly. Skip evals on changes that cannot affect behavior (path filters). Run only the slices whose prompts or tools changed, with the full suite nightly as the safety net. Use one epoch on must-pass smoke cases and save repeats for the statistical gate. Cache judge calls on unchanged outputs. Swap in a cheaper judge model only after checking it agrees with your humans as well as the expensive one (Module 13). There is no standard "right" eval spend; compare it with the cost of one bad release reaching users, which is usually the more persuasive number for a budget holder.

Review cadence

Evals decay without a rhythm. A cadence that works for many teams:

Reporting to stakeholders

Executives do not need your eval log. They need to know whether quality is good enough, whether it is moving, what is wrong, and what decision is needed. The rules that make a report trustworthy:

• Every number with its n and its confidence interval. Worked example: 420 of 500 randomly sampled traces pass, so p = 0.84. The standard error is √(0.84 × 0.16 / 500) ≈ 0.016, and the 95% interval is 0.84 ± 1.96 × 0.016, about 81% to 87%. (Module 04 covers intervals.)
• Compare with a target and with last period, honestly. If the target is 85% and the interval is 81% to 87%, you cannot say you missed it, or that you met it. Say exactly that. Distinguishing 84% from 85% would need many thousands of samples, so set targets as bands and act on clear movements.
• Show the slices that matter. An 84% average can hide a 71% slice that is the most expensive to get wrong.
• Translate into business terms. "Tickets resolved without a human", "cost per resolved ticket", "incidents traced to the agent", next to the quality number.
• Say how the number was made. Which judge, when it was last calibrated against humans, which dataset version. One line is enough, but its absence should make readers suspicious.
• End with the decision. What you recommend and what you need from the reader.

Eval debt and refresh

Like technical debt in code (Sculley et al. described "hidden technical debt" in ML systems in 2015), eval suites accumulate eval debt: small shortcuts and neglect that make the numbers less meaningful every month. Symptoms to look for:

• Saturation. Everything passes every run. The suite still catches regressions, but it no longer tells you whether a change made things better.
• Staleness. The product, policies or tools changed and old cases now test behavior nobody wants (the refund window changed from 30 to 60 days, and 40 cases still expect 30).
• Mix drift. The suite's mix of slices no longer resembles production, so the headline number is weighted toward yesterday's users.
• Uncalibrated judges. The judge prompt or judge model changed, or production outputs changed style, and nobody re-measured agreement with humans.
• Tolerated noise. Cases that flip between runs for environment reasons (timeouts, sandbox errors) and have been ignored for so long that real failures are ignored too.
• Orphans and leakage. Evals nobody owns; cases copied into prompts or few-shot examples, so the system is being tested on its own answer key.

A refresh policy turns this into routine. A common shape: every month, add cases mined from the last month's production failures and review queue; every quarter, move saturated cases to the regression archive, rewrite or delete stale ones, re-weight slices to match production, and recalibrate judges on a fresh human-labeled sample. Version the dataset (refund-suite v14) and print the version next to every number, because a score on v14 is not comparable with a score on v9. The monthly suite-health step can be a short script over your run history:

"""Suite health: flag saturated, always-failing and noisy cases from run history.
history: one row per (case_id, run_id) of the SAME baseline system, passed in {0, 1},
and error_type (None, "timeout", "sandbox", ...) for environment failures."""
import pandas as pd

def suite_health(history: pd.DataFrame, last_n_runs: int = 20) -> pd.DataFrame:
    runs = history["run_id"].drop_duplicates().sort_values().tail(last_n_runs)
    h = history[history["run_id"].isin(runs)]
    stats = h.groupby("case_id").agg(
        pass_rate=("passed", "mean"),
        runs=("passed", "size"),
        env_errors=("error_type", lambda s: s.notna().mean()),
    )

    def status(r) -> str:
        if r.runs < 5:            return "too new"
        if r.env_errors > 0.2:    return "noisy: fix environment"   # not the agent's fault
        if r.pass_rate == 1.0:    return "saturated: archive?"
        if r.pass_rate == 0.0:    return "always fails: broken case or capability target?"
        return "active"          # between 0 and 1: informative (agents are stochastic)

    stats["status"] = stats.apply(status, axis=1)
    return stats.sort_values(["status", "pass_rate"])

# print(suite_health(history)["status"].value_counts())

Note the design choice: on an agent, a case that passes 60% of the time is not "flaky" in the software-testing sense; it is measuring real variance, and it is often the most informative case you have (Module 04). Only failures caused by the environment, not the agent, belong in the "noisy" bucket.

Check yourself

Sources and further reading

• OpenTelemetry, Inside the LLM Call: GenAI Observability with OpenTelemetry (May 2026) and the GenAI semantic conventions repository.
• Dash0, OpenTelemetry GenAI semantic conventions explained (stability status, span names, content capture).
• Langfuse docs, Python SDK instrumentation and scores via SDK.
• promptfoo, GitHub Action, command line (PROMPTFOO_PASS_RATE_THRESHOLD, exit code 100) and CI/CD integration.
• UK AISI / Meridian Labs, Inspect: eval logs (read_eval_log, log formats).
• Anthropic, Demystifying evals for AI agents (Jan 2026) and Writing effective tools for agents (Sept 2025).
• Evan Miller, Adding Error Bars to Evals (paired differences, clustered errors, power analysis).
• OpenAI, Evaluation best practices (continuous evaluation, anti-patterns).
• Hamel Husain, Your AI Product Needs Evals and Evals FAQ.
• Eugene Yan, Evaluating the Effectiveness of LLM-Evaluators (small evaluators for guardrails).
• Shankar et al., Who Validates the Validators? (criteria drift).
• Hamel Husain, A Field Guide to Rapidly Improving AI Products (the "benevolent dictator" quality owner, review habits).
• Sculley et al. (Google), Hidden Technical Debt in Machine Learning Systems, NeurIPS 2015, and Breck et al., The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction, 2017 (maturity rubrics and debt, for classic ML).