Evaluating agent skills

A skill is a folder of know-how that an agent loads only when it thinks the skill is relevant. That single design choice creates two separate ways to fail: the skill never fires when it should, or it fires and does not actually help. This module shows how to measure both, following the method in Anthropic's own open-source skill-creator skill, and then goes further: adherence, many skills at once, portability across models, and skills as a security supply chain.

What a skill is

Imagine you hire a very capable generalist. They can write, code and analyze data, but they do not know your company's way of doing things: which report template finance expects, which chart colors are on-brand, which validation step always catches the broken export. You could brief them on all of it every morning, but most days they would not need most of it. A better approach is a shelf of labeled binders. The generalist glances at the labels, pulls down the right binder when a task calls for it, and only then reads the details.

An agent skill is one of those binders. Anthropic introduced Agent Skills in October 2025. Concretely, a skill is a folder containing:

• A file called SKILL.md. It starts with YAML frontmatter (a small block of key: value lines between --- markers) holding at least a name and a description, followed by Markdown instructions (the body).
• Optional bundled resources: scripts/ (executable code for deterministic or repetitive steps), references/ (longer documents loaded into context only when needed) and assets/ (files used in the output, such as templates, icons or fonts).

Two limits from the spec matter for evals. The name can be at most 64 characters and the description at most 1,024 characters; skill-creator's validator enforces both and its description optimizer aims for roughly 100 to 200 words. That short description is everything the agent knows about the skill before deciding to open it, so it carries almost all of the triggering behavior. Hold on to that thought: it is why trigger evals exist.

Progressive disclosure: three levels of loading

The key design idea is progressive disclosure: show the model a little, and let it ask for more. Skills load in three levels:

1. Level 1, metadata. The name and description of every installed skill sit in the context all the time. skill-creator puts this at about 100 words per skill.
2. Level 2, the body. When the agent decides a skill is relevant, the whole SKILL.md body is loaded. The guidance is to keep it under about 500 lines, and to split anything bigger into reference files with clear pointers.
3. Level 3, bundled resources. Reference files are read only when a step needs them. Scripts can be executed without their code ever entering the context: only their output does. This level is effectively unlimited.

Skill vs tool vs MCP server vs system prompt

Beginners often blur these four. They sit in different places around the agent loop and fail in different ways, so they need different evals. This module covers skills; tools and MCP servers get their own treatment in Module 24.

Why skills need their own evals

A normal prompt change has one job: make the output better. A skill has two independent jobs, and skill-creator frames its whole method around them:

1. Does it trigger at the right times? The agent must decide, from the description alone, to open the skill for relevant requests and to leave it closed for irrelevant ones.
2. When it triggers, does it improve the output? A skill that loads perfectly but adds nothing (or makes things worse) is just context bloat.

These are independent. A brilliant body behind a vague description never gets read. A crisp description in front of a muddled body gets read and then ignored. Measuring only the end result mixes the two, and you cannot tell which half to fix.

So a complete skill eval has at least three measurements, plus two situational ones:

Part 1: Trigger evals

Triggering is a binary classification problem. Each query is a case. The label is "should this skill fire?" and the prediction is "did it fire?". Everything you learned about precision and recall in Module 04 applies directly.

How triggering actually works

Skills appear in the agent's list of available skills with just their name and description, and the model decides whether to consult one based on that description. skill-creator adds two observations that shape how you write trigger evals:

• Claude tends to under-trigger skills, that is, not use them when they would help. Its advice is to make descriptions "a little bit pushy": say what the skill does and list the situations where it should be used, even when the user does not name the skill.
• Simple, one-step queries may not trigger a skill even when the description matches perfectly, because the agent can handle them directly with basic tools. "Read this PDF" is a poor trigger test. Complex, multi-step or specialized queries trigger reliably when the description matches.

The second point is subtle and important: a trigger eval full of short, easy queries will show low recall for every description, and you will waste time "fixing" a description that is fine. Your queries must be substantive enough that a skill would genuinely help.

Building the query set

skill-creator recommends about 20 queries: 8 to 10 that should trigger and 8 to 10 that should not. The format is a plain JSON list:

[
  {"query": "ok so finance just dropped q3_sales_FINAL.csv in /data and the CFO wants the usual quarterly writeup by thursday, same format as last quarter pls", "should_trigger": true},
  {"query": "Can you prepare the Q2 business review document for EMEA? Revenue by country is in the 'Q2' tab of the attached sheet; flag anything down more than 10% QoQ.", "should_trigger": true},
  {"query": "what was our Q3 revenue growth vs Q2? just the percentage, csv is data/q3_sales.csv", "should_trigger": false},
  {"query": "write a python job that pulls the quarterly CSV exports from s3 every night and loads them into postgres", "should_trigger": false}
]

The guidance on writing these is specific, and worth following closely:

• Realistic, concrete and detailed. File paths, column names, company context, a little backstory, casual phrasing, even typos. Bad: "Create a chart". Good: a paragraph that sounds like a real person with a real file.
• Should-trigger coverage. Different phrasings of the same intent (formal and casual), cases where the user never names the skill or file type but clearly needs it, uncommon use cases, and cases where this skill competes with another and should win.
• Should-not-trigger = near-misses. The most valuable negatives share keywords or concepts with the skill but actually need something else: adjacent domains, ambiguous phrasing where naive keyword matching would fire. "Write a fibonacci function" as a negative for a report skill is too easy and tests nothing. In the example above, both negatives mention quarterly CSVs, but one only needs a number and the other needs a data pipeline.

Repeats and the trigger rate

Triggering is stochastic: the same query can fire on one run and not on the next. So each query is run 3 times, and the result per query is a trigger rate (fired runs divided by total runs). In skill-creator's scripts a query "passes" if its trigger rate is at least a threshold (default 0.5) when it should trigger, or below that threshold when it should not.

How does the harness know a skill fired? skill-creator's run_eval.py installs the description temporarily, runs the query headlessly through the Claude Code CLI (claude -p), and watches the streamed events: if the agent's first tool call is a Skill or Read call that names this skill, it counts as triggered; any other first tool call counts as not triggered. That is a precise, cheap, code-based grader (Module 03), which is exactly what you want for a classification eval.

The confusion matrix, worked

Suppose you run 10 should-trigger and 10 should-not-trigger queries, 3 times each: 60 runs. You can count at the run level:

At the query level, with the 0.5 threshold, the same data might look like this: of the 10 positives, 8 fired in at least 2 of 3 runs (pass) and 2 fired in 1 or 0 runs (fail); of the 10 negatives, 9 fired in at most 1 run (pass) and 1 fired in 2 runs (fail). Score: 17 of 20 queries pass. The two views answer different questions. Run-level rates tell you how often a real user would see the behavior; query-level pass counts tell you which phrasings are broken, which is what you need to fix the description.

What a good description looks like

Because the description is the only thing the agent sees before deciding, description quality is trigger quality. Before running any optimizer, check a description against the rules that skill-creator and its scripts encode:

A description review like this is cheap and catches the obvious problems, but it is a hypothesis, not a measurement. Only the trigger eval tells you whether the description actually works with a given model.

Try it: move the description's trigger threshold and watch precision and recall trade off.

Optimizing the description without fooling yourself

Once you can score a description, the obvious next step is to improve it automatically: ask a model to rewrite the description based on which queries failed, re-score, repeat. skill-creator has a script for exactly this (run_loop.py), and its design contains a classic machine-learning lesson.

If you tune the description against the same 20 queries you score it on, you will overfit: the description slowly turns into a list of those exact queries' keywords, scores 20 of 20, and then fails on the next real user. So the loop:

1. Splits the set into 60% train and 40% held-out test, stratified so both halves contain should-trigger and should-not-trigger queries (with 10 and 10, that is 6 + 6 train and 4 + 4 test).
2. Evaluates the current description on both halves, running each query 3 times.
3. Asks Claude to propose an improved description based on the train failures only.
4. Repeats for up to 5 iterations.
5. Returns the best_description, selected by test score, not train score.

It also runs the eval with the same model that powers the user's session, so the triggering test matches what the user will actually experience.

Part 2: Uplift evals

Now the second question: when the skill is used, is the output better? "Better" must mean better than something. The heart of an uplift eval is a paired comparison: the same task run with the skill and against a baseline.

Choosing the baseline

• New skill: the baseline is no skill at all. Same prompt, same model, same tools, no skill path. This answers "is the skill worth having?".
• Improving an existing skill: the baseline is the old version. Snapshot the skill folder before editing and point the baseline run at the snapshot. This answers "is the new version better than the old?".

skill-creator insists on one operational detail: spawn the with-skill and baseline runs in the same turn, at the same time, rather than running all with-skill runs first and baselines later. Running them together means both arms see the same model endpoint, load and conditions, and it avoids the temptation to stop once the with-skill results look good. It is the agent-eval version of running control and treatment together in an experiment.

The test cases: evals.json

Test prompts live in evals/evals.json inside the skill folder. Start with 2 or 3 realistic prompts, and add verifiable statements (the expectations, also called assertions) while the first runs are in progress:

{
  "skill_name": "quarterly-report",
  "evals": [
    {
      "id": 1,
      "prompt": "finance dropped the Q3 export in data/q3_sales.csv, can you turn it into our usual quarterly report? regions are col B, revenue col E",
      "expected_output": "A .docx built from the company template with a revenue-by-region chart and a short executive summary",
      "files": ["evals/files/q3_sales.csv"],
      "expectations": [
        "The report is built from assets/template.docx (same headers and footer)",
        "Revenue-by-region totals in the chart match the CSV to the dollar",
        "The executive summary names the highest and lowest region by revenue",
        "The chart was produced by scripts/make_chart.py"
      ]
    }
  ]
}

What makes a good assertion, according to skill-creator:

• Objectively verifiable. Someone else (or a script) could check it and agree. "Totals match the CSV to the dollar" is checkable; "the report is professional" is not.
• Descriptively named. Someone glancing at the results should understand what each one checks without opening anything.
• Not forced onto subjective qualities. Writing style or design quality go to human review instead. Forcing a fuzzy judgment into a pass/fail assertion gives you a noisy number you will over-trust.
• Scripted when possible. For assertions a program can check (file exists, totals match, template headers present), write a script instead of eyeballing: faster, more reliable, reusable across iterations. This is the code-grader-first rule of Module 03.

The pipeline

The grader: evidence, not vibes

skill-creator's grader (agents/grader.md) is an LLM-as-a-judge (Module 13) with an unusually strict brief. It reads the full transcript and examines the output files itself (it is told not to trust what the transcript says was produced), then marks each assertion pass or fail. A few of its rules are worth memorizing because they apply to any grader:

• Pass requires genuine substance. "A file exists with the right name but wrong or empty content" is a fail. So is meeting an assertion "by coincidence rather than by actually doing the work".
• The burden of proof is on the assertion. When uncertain, fail.
• No partial credit. Each assertion is pass or fail.
• Extract and verify claims beyond the assertions: factual ("the form has 12 fields"), process ("used pypdf") and quality claims, flagging any that cannot be verified.
• Critique the evals themselves. "A passing grade on a weak assertion is worse than useless: it creates false confidence." If an assertion would also pass for a clearly wrong output, or an important outcome has no assertion, the grader says so.

The result is a grading.json per run. Its expectations array must use the field names text, passed and evidence, because the viewer depends on them:

{
  "expectations": [
    {
      "text": "Revenue-by-region totals in the chart match the CSV to the dollar",
      "passed": true,
      "evidence": "Ran check_totals.py: all 6 regions match (APAC 1,204,310 = CSV sum)"
    },
    {
      "text": "The executive summary names the highest and lowest region by revenue",
      "passed": false,
      "evidence": "Summary names APAC as highest but calls LATAM lowest; CSV shows MEA is lowest"
    }
  ],
  "summary": {"passed": 3, "failed": 1, "total": 4, "pass_rate": 0.75},
  "claims": [
    {"claim": "All six regions are included", "type": "factual", "verified": true,
     "evidence": "Six rows in the chart data table"}
  ],
  "eval_feedback": {
    "suggestions": [
      {"assertion": "The report is built from assets/template.docx (same headers and footer)",
       "reason": "Passes in baseline runs too when the model copies a generic header; check the footer text exactly"}
    ],
    "overall": "Totals are well checked; the template assertion is weak."
  }
}

Aggregate, then look past the aggregate

An aggregation script (aggregate_benchmark.py) turns all the grading files into a benchmark.json with, for each configuration, the mean, standard deviation, min and max of pass rate, time and tokens, plus the delta between configurations. Time and tokens come from a timing.json saved for each run as soon as it finishes (the notification that carries total_tokens and duration_ms is the only chance to capture them).

Then comes a step many teams skip: the analyst pass (agents/analyzer.md, "Analyzing Benchmark Results"). Its job is to surface patterns that aggregate numbers hide, not to suggest fixes. For each assertion across all runs, it asks:

• Does it always pass in both configurations? Then it does not differentiate skill value (a non-discriminating assertion).
• Does it always fail in both? It may be broken, or beyond the model's ability.
• Does it always pass with the skill and fail without? That is where the skill clearly adds value.
• Does it always fail with the skill but pass without? The skill may be hurting.
• Is it highly variable? Possibly a flaky assertion or non-deterministic behavior.

It also compares time, tokens and tool calls across configurations (does the skill make runs much slower or more expensive?) and looks for outlier runs that skew averages.

Worked example: the quarterly-report skill

Four test prompts, 3 runs each per configuration (a new skill, so the baseline is no skill). Here is what the benchmark summary and a per-assertion breakdown might look like. These numbers are illustrative, chosen to show the patterns the analyst pass is meant to catch.

The analyst notes might read:

[
  "Assertion 'Output is a .docx file' passes 100% in both configurations - may not differentiate skill value",
  "Eval 'emea-multi-currency' shows high variance with skill (1.00, 0.40, 1.00) - run 2 converted EUR twice; may be flaky",
  "Without-skill runs never use the company template (0/12)",
  "Skill adds 18s and ~2,600 tokens per run on average but improves pass rate by 0.42"
]

Three expert lessons hide in this small example:

1. Some assertions inflate uplift by construction. "Chart made by make_chart.py" and "built from the company template" can never pass without the skill, because the baseline has no access to those files. They are legitimate checks, but they measure "did the agent use the skill?" (adherence), not "is the result better?". Report them separately, or the headline delta will look bigger than the real quality gain. The honest uplift number comes from outcome assertions both arms could pass (totals correct, summary correct).
2. Cost is part of the result. A skill that adds 50% tokens and 18 seconds for a +0.42 pass-rate gain is a great trade. The same cost for a +0.03 gain probably is not. Always read the three columns together.
3. A 4-prompt suite is a smoke test. skill-creator itself says to expand the test set and try again at larger scale once the skill looks good. To claim uplift with confidence you need enough prompts for a confidence interval, which is the Python section below.

The human review loop and iterations

Numbers alone do not tell you why an output is worse. skill-creator generates a viewer with two tabs: Outputs (one test case at a time: prompt, output files rendered inline, previous iteration's output, formal grades, and a feedback box) and Benchmark (the stats). The reviewer's comments are saved to feedback.json; empty feedback means "looked fine". It even says to generate the viewer before the agent evaluates the outputs itself, so a human sees real outputs as early as possible.

The advice on how to improve is as much eval wisdom as skill-writing wisdom:

• Generalize from the feedback. You are iterating on a handful of examples, but the skill will be used on huge numbers of different prompts. Fiddly, overfitted fixes that only rescue these test cases are useless. This is the uplift-eval twin of the train/test lesson in Part 1.
• Read transcripts, not just outputs. If the skill makes the agent waste time on unproductive steps, remove the part of the skill causing it and see what happens.
• Look for repeated work. If every test run independently wrote a similar helper script, bundle that script in scripts/ once.
• Explain the why. ALWAYS and NEVER in capitals are a yellow flag; explaining the reasoning usually works better with capable models.

Part 3: Adherence and process

An uplift number tells you the outcome improved. It does not tell you the skill is why. Maybe the agent loaded the skill, skimmed it, and solved the task its own way; the uplift came from something else, and the next model update will erase it. Adherence evals check the trajectory, the sequence of tool calls, against what the skill says to do. This is process grading from Module 05, applied to skills.

Typical adherence checks, most of them simple code over the transcript:

Two cautions. First, do not demand rigid adherence where it does not matter: if the agent reached the right result by a reasonable different path, that may be fine, and over-strict process checks punish good agents (Module 05 discusses this trade-off). Reserve hard process checks for steps that exist for a reason: validation, safety, using the canonical script so numbers are consistent. Second, adherence failures are information about the skill, not only the agent. skill-creator's analyzer scores instruction-following and lists concrete issues ("did not use the skill's formatting template", "invented own approach instead of following step 3") precisely so you can rewrite the step that was misread.

Part 4: Many skills installed

A real agent rarely has one skill. It may have dozens: documents, spreadsheets, slides, brand guidelines, several team-specific workflows. Now the trigger question becomes a multi-class selection question: given this request, which skill (if any) should load? Your skill's triggering, measured alone, can look perfect and still fail in a crowd.

How to evaluate selection

1. Build a labeled routing set. Queries labeled with the one skill that should load, or "none". Include queries for every skill you ship, plus near-misses that belong to none.
2. Install a realistic set of distractors. Run with the full set of skills your users actually have, not just yours. If you do not know, test at several sizes (for example 5, 20, 50 skills) and plot the curve.
3. Measure multi-class metrics. Top-1 selection accuracy, per-skill precision and recall, and a skill-by-skill confusion matrix. The off-diagonal cells show exactly which pairs collide.
4. Hunt collisions before users do. Compare descriptions pairwise (even simple embedding similarity works as a first filter) and write disambiguation queries for the most similar pairs: "board slides from Q3 numbers" should go to board-deck, "the Q3 report doc" to quarterly-report.
5. Watch for double loads. Sometimes the agent loads two skills whose instructions conflict (two different chart styles). Count runs with more than one skill loaded and check the output against both.

Part 5: Portability and regressions

Because the format is an open standard, the same skill folder may run under different models (a large model and a small, cheap one) and different harnesses (a coding CLI, an IDE, a chat app, your own agent built on an SDK). Each combination can change both halves of the funnel: triggering depends on how the harness presents the skill list and on the model's judgment; uplift depends on how capable the model already is without the skill.

Read the table as a pattern, not data: a smaller model might trigger less reliably yet gain more from the skill when it does fire, because it lacked the know-how to begin with. A different harness might present skills differently and produce more false triggers. You only find out by running the same suite on each target, which is why skill-creator says to run trigger evals with the model the user will actually use.

When the model updates

Skills are written against a model's current behavior, and models change. Treat every model update as a reason to rerun the skill suite (Module 15 covers wiring this into CI). Three things can move:

• Triggering drifts. A new model may be more or less eager to consult skills. Rerun the trigger set and compare rates per query, not just the total.
• Uplift shrinks. If the new baseline model already does the task well, the skill's delta may drop toward zero. That is not a regression in quality; it is a signal the skill may now be dead weight (context cost for no gain) and could be slimmed down.
• Uplift turns negative. Instructions written to steer a weaker model ("always do X in exactly this way") can hold back a stronger one. An assertion that "always fails with the skill but passes without" is the analyst's flag for this.

Keep the pinned baseline idea from Part 2: store a snapshot of each released skill version and its benchmark, so a later rerun can separate "the model changed" from "the skill changed".

Part 6: Safety, skills as a supply chain

A skill is instructions plus, often, executable code, and the agent runs that code with whatever permissions it has. Installing a third-party skill is therefore closer to installing a software package than to reading a document. skill-creator states a "Principle of Lack of Surprise": a skill must not contain malware, exploit code or anything that could compromise security, and its contents should not surprise the user in their intent if described. Your evals should verify that, not assume it. Module 17 covers safety evals in general and Module 25 covers agent security evals in depth (prompt injection, tool poisoning, sandbox escape and more). This section only covers what is specific to skills.

How to evaluate skill safety

• Static review. Read every script and reference file, not just SKILL.md. Flag network calls, reads of credentials or home directories, obfuscated code, instructions that tell the agent to hide actions from the user, and remote content fetched at run time (which can change after review). Re-review every update as a diff.
• Sandboxed dynamic runs. Run the skill's own test prompts in an isolated environment seeded with canary secrets (fake keys that should never leave) and log all outbound traffic and file access. Any canary in egress is a hard fail, the same canary idea as in Module 17.
• Adversarial skills. Build a small suite of deliberately malicious test skills (in a sandbox, with harmless placeholder payloads): a skill whose body asks the agent to send a file to an external URL, a skill whose script tries to read environment variables, a skill with an over-broad description. Measure two things: does your review pipeline flag them (detection rate), and does the agent comply, refuse or warn the user when it loads them (compliance rate)? A good agent should treat a skill's instructions as less trusted than the user's when they conflict with the user's intent.
• Trigger hijacking. Run a malicious skill's description through your routing eval (Part 4). A description engineered to load on everything shows up as a very high false-trigger rate.
• Least privilege as a backstop. Evals miss things. Pair them with runtime limits: no production credentials in the agent's environment, network egress allow-lists, confirmation before irreversible actions. Module 25 shows how to measure attack success rates and utility under attack for these defenses.

What research says

Skills are young, so the research base is small. The most relevant benchmark so far is SkillsBench ("Benchmarking How Well Agent Skills Work Across Diverse Tasks", arXiv 2602.12670, February 2026). Its design is exactly the uplift eval of Part 2 at scale: the same tasks, run by agents with and without skills. The paper reports:

• About 84 tasks across 11 domains.
• Curated, human-written skills raised the pass rate by roughly 16 percentage points on average, with large variation by domain: small gains in software engineering, where models already know a lot, and much larger gains in some specialized domains such as healthcare.
• Self-generated skills, where the model writes its own skill before solving the task, gave no reliable benefit on average.
• Focused skills beat long, comprehensive ones.

Each finding maps onto an eval lesson. Variation by domain means an average uplift hides a lot, so report per domain or per task family. The self-generated result means "the model can write its own skills" is a claim to test, not assume; the know-how has to come from somewhere. And the focused-beats-comprehensive result echoes skill-creator's advice to keep the body lean and push detail into reference files: more context is not automatically more help.

The math in Python

Two small functions cover the core numbers of this module. The first computes run-level trigger metrics and query-level pass counts from repeated runs, using the same pass rule as skill-creator (trigger rate compared with a 0.5 threshold). The second computes a paired bootstrap confidence interval for uplift. "Paired" matters: each eval is run in both arms, so you resample whole evals and keep each with-skill score next to its own baseline. That removes the "some evals are just harder" noise both arms share (Module 04 explains why paired differences are so much tighter).

import random
from statistics import mean

# ---- Part 1: trigger metrics from repeated runs -------------------------
# One record per query; "fired" holds one entry per repeat (3 runs each).
trigger_results = [
    # should-trigger queries (10)
    *[{"should_trigger": True, "fired": f} for f in [
        [1,1,1],[1,1,1],[1,1,1],[1,1,1],[1,1,1],
        [1,1,0],[1,0,1],[0,1,1],[1,0,0],[0,0,0]]],
    # should-not-trigger queries (10), mostly near-misses
    *[{"should_trigger": False, "fired": f} for f in [
        [1,1,0],[0,1,0],[0,0,1],[0,0,0],[0,0,0],
        [0,0,0],[0,0,0],[0,0,0],[0,0,0],[0,0,0]]],
]

def trigger_metrics(results, threshold=0.5):
    """Run-level confusion matrix plus query-level pass count."""
    tp = fn = fp = tn = 0
    query_pass = 0
    for r in results:
        fired = [bool(x) for x in r["fired"]]
        k, n = sum(fired), len(fired)
        if r["should_trigger"]:
            tp += k; fn += n - k
            query_pass += (k / n) >= threshold      # same rule as skill-creator
        else:
            fp += k; tn += n - k
            query_pass += (k / n) < threshold
    precision = tp / (tp + fp) if tp + fp else float("nan")
    recall = tp / (tp + fn) if tp + fn else float("nan")
    f1 = 2 * precision * recall / (precision + recall)
    return {"tp": tp, "fn": fn, "fp": fp, "tn": tn,
            "precision": round(precision, 3), "recall": round(recall, 3),
            "f1": round(f1, 3), "false_trigger_rate": round(fp / (fp + tn), 3),
            "queries_passed": f"{query_pass}/{len(results)}"}

print(trigger_metrics(trigger_results))

# ---- Part 2: paired bootstrap CI for uplift -----------------------------
# Mean pass rate per eval (averaged over its 3 runs), same 12 evals in both arms.
with_skill = [0.89, 1.00, 0.78, 0.89, 1.00, 0.67, 0.89, 0.78, 1.00, 0.89, 0.56, 0.89]
baseline   = [0.44, 0.67, 0.33, 0.56, 0.78, 0.44, 0.33, 0.56, 0.89, 0.44, 0.56, 0.22]

def paired_bootstrap(a, b, n_boot=10_000, seed=0):
    """95% CI for mean(a - b), resampling whole evals (keeps the pairing)."""
    diffs = [x - y for x, y in zip(a, b)]
    rng = random.Random(seed)
    boots = sorted(
        mean(rng.choice(diffs) for _ in diffs) for _ in range(n_boot)
    )
    lo, hi = boots[int(0.025 * n_boot)], boots[int(0.975 * n_boot) - 1]
    return round(mean(diffs), 3), (round(lo, 3), round(hi, 3))

print(paired_bootstrap(with_skill, baseline))

Output (illustrative data):

{'tp': 22, 'fn': 8, 'fp': 4, 'tn': 26, 'precision': 0.846, 'recall': 0.733, 'f1': 0.786, 'false_trigger_rate': 0.133, 'queries_passed': '17/20'}
(0.335, (0.232, 0.438))

The uplift is +0.335 with a 95% interval of roughly +0.23 to +0.44. Because the whole interval is above zero, you can say the skill helps on this suite. Notes for real use:

• Resample evals, not runs. The three runs of one eval are correlated (same prompt, same difficulty). Treating 36 runs as 36 independent points would make the interval falsely narrow. This is the clustering point from Module 04.
• A bootstrap over 3 or 4 evals is meaningless. You need something like a dozen or more evals before an interval says much. Small suites are for iteration; bigger suites are for claims.
• Exclude assertions the baseline cannot pass by construction (Part 2) before computing uplift, or compute two numbers: outcome uplift and adherence.
• Same interval idea for triggering. To compare two descriptions, compute the per-query difference in trigger rate and bootstrap over queries.

A skill eval checklist

Check yourself

Sources and further reading

• Anthropic, anthropics/skills on GitHub, especially skill-creator: SKILL.md, agents/grader.md, agents/analyzer.md, agents/comparator.md, references/schemas.md and scripts/run_eval.py / run_loop.py. The primary source for the method in Parts 1 and 2.
• Anthropic Engineering, Equipping agents for the real world with Agent Skills (October 2025).
• Claude Docs, Agent Skills overview and Skill authoring best practices.
• Agent Skills open standard: specification and agentskills/agentskills on GitHub.
• Li et al., SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks (February 2026), and the SkillsBench site.
• Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forward (survey, 2026).
• Snyk, ToxicSkills: a study of agent skills supply-chain compromise (2026).
• Philipp Schmid, Practical Guide to Evaluating and Testing Agent Skills (2026), and his AI Engineer talk Don't Ship Skills Without Evals.
• Evan Miller, Adding Error Bars to Evals (2024), for paired and clustered intervals.