Evaluating tools and MCP servers

An agent is only as good as the tools it is handed. A vague description, a confusing parameter, a response that dumps 20,000 tokens of JSON, or a server that quietly changes after you approved it can all sink an agent that would otherwise succeed. This module turns the camera around: instead of holding the tools fixed and grading the agent, we grade the tools, the MCP servers that ship them, and the whole selection problem that appears when an agent has hundreds of tools to choose from.

Where this module fits

Module 06 taught you to grade an agent's tool use: the function-calling loop, BFCL-style AST matching of a single call, subset matching of a chain, and τ-bench's end-state grading with pass^k. It closed with a short section, "Evaluating the tools themselves". This module starts where that section stops. We keep the same graders, but change what is under test.

Siblings in Part V cover the other building blocks: the harness around the model (Module 21), skills, which are packaged know-how the agent chooses to load (Module 22), and memory (Module 23). Tool security gets a short, practical treatment here; Module 25 goes deep on attacks and red-teaming.

What a tool and an MCP server are

Start with the smallest unit. A tool is a function the agent may ask to run. To the model, a tool is nothing but three pieces of text: a name (search_contacts), a description in plain English, and an input schema in JSON Schema (the standard format for saying "this argument is a string, this one an integer from 1 to 20, this one is required"). The model never sees your code. Everything it knows about when and how to use the tool comes from those three pieces of text, plus whatever the tool returns when called.

The three roles: host, client, server

MCP defines three roles. Learn them precisely, because failures (and blame) sit at the boundaries between them.

• Host: the application the user runs, such as Claude Desktop, Claude Code, an IDE, or your own agent app. It owns the model, the conversation, the user's consent decisions and the security policy.
• Client: a connector inside the host. The host creates one client per server; each client keeps a single stateful session with its server and handles the protocol for it.
• Server: a program that exposes capabilities. It may run locally as a subprocess, talking over stdio (standard input and output), or remotely over Streamable HTTP. Messages in both cases are JSON-RPC 2.0: small JSON objects with a method, params and an id.

The three server primitives: tools, resources, prompts

A server can offer three kinds of things, and they differ in who decides when they are used:

• Tools are model-controlled: the model decides to call them (tools/list to discover, tools/call to run). Tools can have side effects. Most evaluation effort goes here.
• Resources are application-controlled: read-only data identified by a URI (a file, a database schema, a ticket) that the host decides to attach to context.
• Prompts are user-controlled: reusable templates the user picks, often as a slash command.

One session, message by message

It helps to see the actual conversation between client and server, because each message is a place you can hook an eval. The session starts with initialize, where the two sides agree on a protocol version and declare their capabilities. Then the client asks tools/list and gets back each tool's name, description, inputSchema, and optionally an outputSchema (the shape of structured results) and annotations, hints such as readOnlyHint and destructiveHint that tell the host whether a tool changes anything. When the model wants a tool, the client sends tools/call; the server replies with content (text, images, links), optional structuredContent, and an isError flag. A server can also announce notifications/tools/list_changed, which means "my tools just changed, list them again". Hold on to that last message: it is useful, and it is also how a rug pull happens.

What can go wrong, layer by layer

When an agent fails a task that uses MCP tools, the fault can live in any of five layers. Good evals are designed so that each layer's failures show up as that layer's failures, not as one undifferentiated "task failed". This is the error-analysis habit from Module 12, applied to the tool stack.

Tool descriptions and schemas are prompts

Here is the single most useful idea in this module: a tool definition is a prompt. The model reads every tool's name, description and schema on every turn, and those words shape its behavior exactly as a system prompt does. Change "Searches contacts." to "Find people by partial name or email; use this first whenever the user names a person but gives no email address", and you have changed the agent's behavior without touching the model, the harness, or a line of the tool's code.

That has two consequences. First, tool definitions deserve the same treatment as prompts: version them, review them, and never change them without re-running evals. Second, you can measure a description. Two descriptions of the same function are two variants in an A/B test, and the metric is how well a fixed agent does with each. Research has started to treat this systematically: one 2026 study titled "MCP Tool Descriptions Are Smelly!" catalogs quality problems ("smells") in real MCP tool descriptions and tests augmented descriptions for agent efficiency, and a companion paper proposes smell-aware evaluation of server descriptions (From Docs to Descriptions). The Berkeley Function Calling Leaderboard added a format-sensitivity study in V4 for the same reason: small changes in how tools and prompts are presented can move scores.

The loop, as an eval procedure

Anthropic's engineering post "Writing effective tools for agents, with agents" (September 2025) describes the loop that Module 06 summarized. Read it here as a precise eval procedure:

1. Prototype. Build a quick version of the tools (wrap them in a local MCP server) and try them yourself with an agent. You will spot rough edges in minutes.
2. Build realistic eval tasks. Tasks grounded in real use, needing several calls, each with a verifiable outcome. A strong task: "Schedule a meeting with Jane next week to discuss our latest Acme Corp project; attach the notes from our last planning meeting and reserve a room." A weak one hands the agent the exact argument ("Schedule a meeting with jane@acme.corp") and tests almost nothing. Avoid graders so strict that they reject valid alternative paths or formats.
3. Run programmatically. A simple agent loop per task (the post suggests direct API calls, with the model asked to output reasoning and feedback blocks before tool calls). Collect: accuracy, total runtime and per-call runtime, number of tool calls, token consumption, and tool error rate.
4. Read transcripts. Metrics tell you that something is off; transcripts tell you what. The post notes that what agents omit in their feedback can matter more than what they include. Lots of redundant calls suggest the tool needs pagination or filters; lots of invalid-parameter errors suggest unclear parameter names or descriptions.
5. Refine, then confirm on a held-out set. Feed the transcripts to an agent (the post uses Claude Code) to propose better tools, apply them, and re-run. Keep a held-out set of tasks that nobody (human or agent) looked at while refining. The post reports that on its own internal Slack and Asana tools, Claude-refined versions beat the human-written ones on held-out tasks. Without a held-out set, you are optimizing descriptions to your dev tasks, which is overfitting with extra steps.

A worked example: before and after

Suppose you ship a small CRM server with search_contacts and send_email. The eval set has 60 tasks such as "Email Jane from Acme that the demo moved to Friday", split into 40 dev and 20 held-out, 5 trials each. You read 30 failing transcripts and find three recurring patterns.

The three patterns from the transcripts, and the fix for each:

Then you re-run both variants on the same tasks and read the dashboard.

Responses and errors are part of the tool too

The post's design advice gives you more things to measure than accuracy. Each piece of advice becomes a metric or a test:

• Token efficiency of responses. Measure tokens per tool response. Add pagination, filtering, sensible default limits and truncation. Anthropic notes that Claude Code limits tool responses to 25,000 tokens by default. The post also suggests a response_format parameter (concise or detailed) and reports that a concise Slack response used roughly a third of the tokens of the detailed one.
• Meaningful context. Return names and human-readable fields rather than opaque UUIDs where you can. Test: count how often the agent makes an extra call only to translate an ID.
• Actionable errors. "Invalid input" teaches nothing; "limit must be an integer from 1 to 20; you sent 'all'" lets the agent fix itself. Test: inject bad arguments and measure recovery rate (the share of errors followed by a successful corrected call).
• Namespacing. With many similar tools, prefixes such as asana_projects_search reduce confusion. Test: a confusion matrix of intended versus chosen tool (next section).
• Consolidation. One schedule_event tool that finds availability and books may beat three low-level tools the agent must chain. Test: calls per task and accuracy, before and after merging.

Scoring a tool call: five levels of correctness

"Did the agent use the tools correctly?" hides five different questions. Separate them and each failure points at a different fix. Module 06 built the single-call AST checker and the ordered-subset chain check; here they become rungs on one ladder, plus two more rungs that matter a lot with real servers.

Precise definitions

• Selection accuracy. Share of decision points where the agent chose the intended tool, or correctly chose none. Report it with a confusion matrix (rows: intended tool; columns: chosen tool; plus a "no call" row and column). Off-diagonal cells name the pairs of tools whose descriptions collide. Include irrelevance cases, where no tool fits, so over-calling is caught.
• Argument correctness in three layers. Schema: required keys present, no unknown keys. Type: an integer is an integer, not "5". Value: after normalization, the value is one of the acceptable ones. The first two can be checked by a JSON Schema validator with no reference answer; the third needs a gold value or an end-state check. Invented values are the dangerous kind, because they pass the first two layers.
• Sequencing. Gold calls appear as an ordered subsequence (extra harmless reads allowed), dependencies hold (verify before refund), and forbidden calls never happen. Report "extra calls" separately as an efficiency metric rather than failing on them.
• Parallel calls. When calls are independent ("weather in Paris, Rome and Oslo"), a capable agent can emit them in one turn. Score completeness (all three made) and batching (made in the same turn). Batching affects latency, not correctness, so weigh it accordingly, and check your host actually supports parallel calls before penalizing their absence.
• Result use. The next action is consistent with what the tool returned. Two cheap automatic checks: every ID the agent passes appeared in some earlier result or in the user's message (provenance), and every isError result is followed by a correction, a retry, or an honest message to the user (error handling). Harder cases need an LLM judge reading the transcript (Module 13).

Tool errors versus agent errors

When a call fails, whose fault is it? This attribution question decides who fixes it: the model or prompt team, or the team that owns the server. It also decides whether the failure should count against the agent in a benchmark at all. A flaky backend that returns HTTP 503 at random is measuring your infrastructure, not the agent (the infrastructure-noise problem from Module 21).

MCP gives you one helpful signal for this. The spec separates protocol errors (a JSON-RPC error, such as an unknown tool name or a malformed request) from tool execution errors, which come back as a normal result with isError: true and a message the model can read and react to. Log both, and do not let a harness swallow either one.

Build it: a scoring harness against a mock server

The code below replays an agent's recorded tool calls against an in-process mock of an MCP server (it mimics tools/list and tools/call result shapes, including isError), then scores every rung of the ladder and attributes each error. It uses the real jsonschema library for the schema and type layer. In a real suite you would record AGENT_RUN from your agent's transcript, or run the agent live against the mock.

# tool_harness.py : replay an agent's tool calls against a mock MCP-style server
# and score selection, arguments, sequencing, parallelism and result use.
import json, re
from jsonschema import Draft202012Validator

# ---------------------------------------------------------------- 1. mock server
TOOLS = [  # what tools/list would return: name, description, inputSchema
    {"name": "find_customer",
     "description": "Find a customer by email. Returns customer_id and name.",
     "inputSchema": {"type": "object", "additionalProperties": False,
                     "properties": {"email": {"type": "string", "format": "email"}},
                     "required": ["email"]}},
    {"name": "list_orders",
     "description": "List a customer's orders, newest first.",
     "inputSchema": {"type": "object", "additionalProperties": False,
                     "properties": {"customer_id": {"type": "string", "pattern": "^C[0-9]+$"},
                                    "limit": {"type": "integer", "minimum": 1, "maximum": 50}},
                     "required": ["customer_id"]}},
    {"name": "get_shipping_status",
     "description": "Get the carrier status for one order.",
     "inputSchema": {"type": "object", "additionalProperties": False,
                     "properties": {"order_id": {"type": "string"}},
                     "required": ["order_id"]}},
    {"name": "refund_order",
     "description": "Refund an order in full. Irreversible.",
     "inputSchema": {"type": "object", "additionalProperties": False,
                     "properties": {"order_id": {"type": "string"},
                                    "reason": {"type": "string"}},
                     "required": ["order_id", "reason"]}},
]
SCHEMAS = {t["name"]: t["inputSchema"] for t in TOOLS}

class MockServer:
    """In-process stand-in for an MCP server: list_tools() and call_tool()."""
    def __init__(self):
        self.customers = {"ana@example.com": {"customer_id": "C42", "name": "Ana"}}
        self.orders = {"C42": [{"order_id": "A17", "total": 59.0},
                               {"order_id": "B02", "total": 12.5}]}
        self.refunded = set()

    def list_tools(self):
        return TOOLS

    def call_tool(self, name, args):
        """Return an MCP-shaped result: {"content": [...], "isError": bool}."""
        def ok(obj):  return {"content": [{"type": "text", "text": json.dumps(obj)}], "isError": False}
        def err(msg): return {"content": [{"type": "text", "text": msg}], "isError": True}
        if name not in SCHEMAS:
            return err(f"Unknown tool: {name}")
        problems = [e.message for e in Draft202012Validator(SCHEMAS[name]).iter_errors(args)]
        if problems:                                   # the server rejects bad input
            return err("Invalid arguments: " + "; ".join(problems))
        if name == "find_customer":
            c = self.customers.get(args["email"].lower())
            return ok(c) if c else err("No customer with that email.")
        if name == "list_orders":
            return ok(self.orders.get(args["customer_id"], [])[: args.get("limit", 10)])
        if name == "get_shipping_status":
            return ok({"order_id": args["order_id"], "status": "in transit"})
        if name == "refund_order":
            known = {o["order_id"] for os_ in self.orders.values() for o in os_}
            if args["order_id"] not in known:
                return err(f"Order {args['order_id']} not found.")
            self.refunded.add(args["order_id"])
            return ok({"refunded": args["order_id"]})

# ---------------------------------------------------------------- 2. replay
def replay(trajectory, server):
    """trajectory: list of turns; each turn is a list of calls made in parallel.
    Returns a flat log with the turn number and the server's result."""
    log = []
    for turn, calls in enumerate(trajectory):
        for c in calls:
            log.append({"turn": turn, **c, "result": server.call_tool(c["name"], c["args"])})
    return log

# ---------------------------------------------------------------- 3. scoring
def norm(v):
    return re.sub(r"\s+", "", v.lower()) if isinstance(v, str) else v

def arg_problems(call, gold):
    """Layered argument check: schema (types, required, unknown keys), then values."""
    out = [f"schema: {e.message}" for e in Draft202012Validator(SCHEMAS[call["name"]]).iter_errors(call["args"])]
    for k, allowed in gold.get("values", {}).items():
        if norm(call["args"].get(k)) not in [norm(a) for a in allowed]:
            out.append(f"value: {k}={call['args'].get(k)!r}, wanted one of {allowed}")
    return out

def score(log, spec):
    names = [c["name"] for c in log]
    report = {}
    # (a) selection: right first tool, and did the set of tools cover the gold set?
    gold_names = [g["name"] for g in spec["gold"]]
    report["first_tool_ok"] = bool(names) and names[0] == gold_names[0]
    report["tool_recall"] = sum(n in names for n in set(gold_names)) / len(set(gold_names))
    # (b) arguments + (c) sequence: gold calls must appear as an ordered subsequence
    i, matched, arg_issues = 0, 0, []
    for g in spec["gold"]:
        while i < len(log) and log[i]["name"] != g["name"]:
            i += 1
        if i == len(log):
            break
        probs = arg_problems(log[i], g)       # scores the FIRST attempt at each gold call
        arg_issues += [f"{g['name']}: {p}" for p in probs]
        matched += 1
        i += 1
    report["sequence_ok"] = matched == len(spec["gold"])
    report["args_ok"] = report["sequence_ok"] and not arg_issues
    report["arg_issues"] = arg_issues
    report["forbidden"] = [n for n in names if n in spec.get("forbidden", [])]
    # (d) parallelism: independent calls should share a turn
    turns = {c["name"]: c["turn"] for c in log}
    report["parallel_ok"] = all(turns.get(a) is not None and turns.get(a) == turns.get(b)
                                for a, b in spec.get("parallel_pairs", []))
    # (e) result use: an ID the agent passed must have appeared in an earlier result
    seen, invented = "", []
    for c in log:
        for k in spec.get("id_args", []):
            if k in c["args"] and str(c["args"][k]) not in seen:
                invented.append(f"{c['name']}.{k}={c['args'][k]}")
        seen += json.dumps(c["result"])
    report["invented_ids"] = invented
    # (f) attribution: who caused each error?
    report["errors"] = [attribute(c) for c in log if c["result"]["isError"]]
    # (g) recovery: every failed call was later retried successfully
    report["recovered"] = all(any(d["name"] == c["name"] and not d["result"]["isError"]
                                  for d in log[j + 1:])
                              for j, c in enumerate(log) if c["result"]["isError"])
    return report

def attribute(call):
    msg = call["result"]["content"][0]["text"]
    if msg.startswith(("Invalid arguments", "Unknown tool")):
        return ("agent", call["name"], msg)       # the call itself was malformed
    if "not found" in msg.lower():
        return ("agent?", call["name"], msg)      # valid shape, wrong value: read the transcript
    return ("tool", call["name"], msg)            # valid call, server failed: fix the server

# ---------------------------------------------------------------- 4. one test case
SPEC = {  # "Ana (ana@example.com) wants her latest order refunded; also where is B02?"
    "gold": [{"name": "find_customer", "values": {"email": ["ana@example.com"]}},
             {"name": "list_orders",   "values": {"customer_id": ["C42"]}},
             {"name": "refund_order",  "values": {"order_id": ["A17"]}}],
    "forbidden": ["delete_customer"],
    "parallel_pairs": [("refund_order", "get_shipping_status")],
    "id_args": ["customer_id", "order_id"],
}
AGENT_RUN = [  # recorded from the agent under test: one list per turn
    [{"name": "find_customer", "args": {"email": "Ana@Example.com"}}],
    [{"name": "list_orders", "args": {"customer_id": "C42", "limit": "5"}}],   # "5" is a string
    [{"name": "list_orders", "args": {"customer_id": "C42", "limit": 5}}],     # retry after error
    [{"name": "refund_order", "args": {"order_id": "A17", "reason": "customer request"}},
     {"name": "get_shipping_status", "args": {"order_id": "B02"}}],           # parallel turn
]

if __name__ == "__main__":
    log = replay(AGENT_RUN, MockServer())
    print(json.dumps(score(log, SPEC), indent=2))

Running it prints this report (abridged to one line per field):

first_tool_ok: true        tool_recall: 1.0        sequence_ok: true
args_ok: false             arg_issues: ["list_orders: schema: '5' is not of type 'integer'"]
forbidden: []              parallel_ok: true       invented_ids: []
errors: [["agent", "list_orders", "Invalid arguments: '5' is not of type 'integer'"]]
recovered: true

Read it rung by rung. Selection, sequence and parallelism are fine. The argument layer failed on the first attempt (a string where the schema wants an integer), the error is attributed to the agent, and the agent recovered on its own. The end state is correct, so an outcome-only grader would score this run as a clean pass and hide a pattern that costs a retry, latency and tokens on every task. Now run 200 recorded trajectories through it and count: if limit type errors show up across several models, the fix belongs in the tool (a clearer description, or a server that accepts "5"), not in the agent.

Two design choices worth copying. First, the harness scores the first attempt at each gold call and reports recovery separately, so "got there eventually" does not hide "got it wrong first". Second, the provenance check (invented_ids) needs no gold answer at all, which makes it cheap to run on production traces as well (Module 15).

Large toolsets: when the agent has hundreds of tools

A single MCP server might expose 5 tools. A developer who connects GitHub, Slack, a database, a browser, a ticket tracker and a cloud console can easily hand the agent over a hundred. Two separate things get worse as that number grows, and you should measure them separately.

1. Context cost. Every tool definition is text the model reads on every request. Anthropic's advanced tool use post describes multi-server setups whose definitions consume tens of thousands of tokens before the agent has done any work. That is money, latency, and context the agent cannot use for the task.
2. Selection difficulty. More tools means more near-duplicates (search_issues in two servers, create_ticket versus create_issue) and more chances to pick wrongly. The RAG-MCP paper names this "prompt bloat" and studies retrieving a small set of relevant tools before the model chooses, and LiveMCPBench asks directly whether agents can "navigate an ocean of MCP tools".

Worked numbers: what the definitions cost

Say six servers expose 60 tools, and a typical definition (name, description, schema with a few parameters) is about 500 tokens. That is 60 × 500 = 30,000 tokens of definitions on every model call. An agent task with 20 model calls re-sends them 20 times: 600,000 input tokens of tool definitions per task, before any conversation or tool results. Prompt caching reduces the price of repeated tokens, but the model still has to attend to them, and they still count against the context window. (Illustrative sizes: measure your own with a token counter, because descriptions vary from 50 to several thousand tokens.)

Tool search and retrieval

The fix most stacks now use is to not load every tool up front. Instead, a retrieval step finds the handful of tools that fit the current request and loads only those. Anthropic's version is the tool search tool: you mark tools with defer_loading, the model sees a search tool (regex or BM25 variants) plus a few always-loaded tools, and it pulls in full definitions on demand. A related pattern, described in Anthropic's code execution with MCP post (November 2025), presents MCP tools as code files the agent can browse and call from a sandbox; the post's example cut token use from about 150,000 to about 2,000 tokens (a 98.7% reduction), because definitions and intermediate results no longer pass through the context.

Retrieval adds a new stage that can fail on its own, so an end-to-end score is no longer enough. Split it.

To measure retrieval you need labels: for each test request, which tools are needed. Then the retrieval metrics from RAG evaluation apply unchanged (Module 06): recall@k, MRR, nDCG. A useful warning from the literature: the ToolRet benchmark paper is titled "Retrieval Models Aren't Tool-Savvy", reporting that retrievers that do well on ordinary document search do poorly at finding the right tool. Test your retriever on tool requests, not on a generic retrieval benchmark.

The distractor scaling test

The most informative large-toolset eval is a scaling curve: fix a task set, then grow the toolset around it and watch selection accuracy. Add distractors in two flavors, because they fail differently: random distractors (unrelated tools from other domains) and near-miss distractors (tools with overlapping names and purposes, like a second search_issues from another tracker). Near-misses are what real users install.

MCP and tool benchmarks

Since 2025 a wave of benchmarks has moved tool evaluation from mock functions (BFCL, ToolBench) toward real MCP servers. They share a direction but differ sharply in how they grade, which is the thing to understand before quoting any score. Every name below was checked against its paper or official page; details are from the papers' abstracts and project pages as of 2026, so check the current versions before relying on a specific count.

Evaluating an MCP server you built

Now suppose you are the one shipping the server: an internal "orders" server your company's agents will use, or a public server for your product. Its users are agents, many of them, on models you do not control. What does a complete eval suite look like? Think of it as the familiar software test pyramid with an extra layer on top for "does it actually help an agent".

Layer 1 and 2: static checks and contract tests

A contract test checks that the server keeps its promises at the protocol level, independent of any model: the tools it advertises have valid schemas and non-empty descriptions, a valid call returns a result of the advertised shape, a bad call returns a readable isError result rather than crashing, and destructive tools are marked as such. For manual poking, the official MCP Inspector is a visual tool for exactly this. For CI, write the checks as tests using the official MCP Python SDK as the client, so you exercise the real protocol path:

# test_orders_server.py : contract tests for an MCP server (pytest + MCP Python SDK)
#   pip install mcp jsonschema pytest anyio ;  pytest -q
import hashlib, json, time
import pytest
from jsonschema import Draft202012Validator
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

SERVER = StdioServerParameters(command="python", args=["orders_server.py"])
APPROVED_MANIFEST = "manifest.sha256"   # committed file: hash of reviewed tool definitions

async def session_run(fn):
    async with stdio_client(SERVER) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            return await fn(session)

def manifest_hash(tools) -> str:
    """Hash exactly what the model will read: names, descriptions, schemas, annotations."""
    canon = [{"name": t.name, "description": t.description,
              "inputSchema": t.inputSchema,
              "annotations": t.annotations.model_dump() if t.annotations else None}
             for t in sorted(tools, key=lambda t: t.name)]
    return hashlib.sha256(json.dumps(canon, sort_keys=True).encode()).hexdigest()

@pytest.mark.anyio
async def test_definitions_are_well_formed():
    tools = (await session_run(lambda s: s.list_tools())).tools
    names = [t.name for t in tools]
    assert len(names) == len(set(names)), "duplicate tool names"
    for t in tools:
        Draft202012Validator.check_schema(t.inputSchema)          # schema itself is valid
        assert t.name.startswith("orders_"), f"{t.name}: not namespaced"
        assert t.description and len(t.description.split()) >= 15, f"{t.name}: description too thin"
        for p, spec in t.inputSchema.get("properties", {}).items():
            assert spec.get("description"), f"{t.name}.{p}: parameter has no description"

@pytest.mark.anyio
async def test_destructive_tools_are_annotated():
    tools = (await session_run(lambda s: s.list_tools())).tools
    for t in tools:
        if any(w in t.name for w in ("delete", "refund", "cancel")):
            assert t.annotations and t.annotations.destructiveHint, f"{t.name}: mark destructiveHint"

@pytest.mark.anyio
async def test_manifest_matches_the_reviewed_version():
    # Any change to what the model reads must be reviewed and re-evaluated.
    tools = (await session_run(lambda s: s.list_tools())).tools
    assert manifest_hash(tools) == open(APPROVED_MANIFEST).read().strip(), \
        "tool definitions changed: rerun the golden-task evals, then update the manifest"

@pytest.mark.anyio
async def test_happy_path_and_output_schema():
    async def go(s):
        tools = {t.name: t for t in (await s.list_tools()).tools}
        t0 = time.perf_counter()
        res = await s.call_tool("orders_get", {"order_id": "A17"})
        return tools["orders_get"], res, time.perf_counter() - t0
    tool, res, secs = await session_run(go)
    assert not res.isError
    if tool.outputSchema:                                         # structured output promised?
        Draft202012Validator(tool.outputSchema).validate(res.structuredContent)
    assert secs < 2.0, f"slow: {secs:.2f}s"

@pytest.mark.anyio
@pytest.mark.parametrize("args", [{"order_id": 17}, {}, {"order_id": "A17", "force": True}])
async def test_bad_input_gives_a_readable_tool_error(args):
    res = await session_run(lambda s: s.call_tool("orders_get", args))
    assert res.isError, "bad input must come back as isError, not succeed silently"
    text = " ".join(c.text for c in res.content if c.type == "text")
    assert "order_id" in text, "error should name the offending parameter so the agent can fix it"

Notice the manifest test. It hashes exactly the text the model reads. Any edit to a description or schema, however small, fails CI until someone re-runs the golden-task evals and approves the new hash. That turns "descriptions are prompts" into an enforced process, and the same idea on the consumer side is your defense against rug pulls (below).

Layer 3: golden tasks through a real agent

Contract tests say the server works; they cannot say an agent can use it. For that, write 30 to 100 realistic tasks (Anthropic's advice from earlier applies), each with a verifiable outcome, and run them with a real agent connected to your server. Inspect can attach an MCP server to an agent directly; its MCP tools documentation shows the pattern:

from inspect_ai import Task, task
from inspect_ai.agent import react
from inspect_ai.dataset import json_dataset
from inspect_ai.tool import mcp_server_stdio, mcp_tools

orders = mcp_server_stdio(name="orders", command="python", args=["orders_server.py"])

@task
def orders_golden(epochs: int = 3):
    return Task(
        dataset=json_dataset("golden_tasks.jsonl"),  # input + metadata for the end-state check
        solver=react(tools=[mcp_tools(orders, tools=["orders_*"])]),  # only this server's tools
        scorer=orders_end_state(),                   # your scorer: query the DB after the run
        epochs=epochs,                               # several trials: report pass^k too
    )

Score the end state as τ-bench does (Module 06), and also collect the tool-level metrics from the harness above: selection confusion, first-attempt argument validity, recovery rate, calls and tokens per task. Run the suite on at least two model families. A server that only works well with one model is fragile, and your users will switch models.

Layer 4 and 5: performance, versioning and regression

• Latency and cost. Record p50 and p95 latency per tool and per task, response size in tokens, and backend cost. Agents call tools many times per task, so a tool that takes 3 seconds turns into a minute of waiting. Add a budget test that fails when p95 regresses by more than, say, 20%.
• Fault injection. Make the backend slow, return 429 or 503, or time out, in a test mode. Measure whether the server returns useful isError messages and whether agents recover instead of looping.
• Versioning. Treat tool definitions as a public API. Renaming a parameter or tightening a schema breaks agents mid-conversation the same way it breaks code. Keep a changelog, version the server, and prefer adding a new tool over changing the meaning of an old one.
• Regression. Keep every past golden-task result. On each release, compare against the last one per task, not just on average, and read the transcripts of tasks that flipped from pass to fail. Rerun the whole suite when a new model version ships, even if the server did not change.

A/B-testing a description in Inspect

Here is the before/after experiment from earlier as a runnable Inspect task. Both variants register a tool under the same name with the same code; only the docstring (which Inspect turns into the description and parameter descriptions the model sees) differs. Run each variant with several epochs, then compare the two logs task by task.

# tool_ab.py : A/B two descriptions of the same tool (Inspect AI)
#   inspect eval tool_ab.py -T variant=before --epochs 5 --model anthropic/claude-sonnet-4-5
#   inspect eval tool_ab.py -T variant=after  --epochs 5 --model anthropic/claude-sonnet-4-5
import json
from inspect_ai import Task, task
from inspect_ai.dataset import Sample
from inspect_ai.model import ChatMessageAssistant
from inspect_ai.scorer import Score, Target, accuracy, mean, scorer, stderr
from inspect_ai.solver import TaskState, generate, system_message, use_tools
from inspect_ai.tool import tool

CONTACTS = [{"name": "Jane Doe", "email": "jane@acme.example", "company": "Acme"},
            {"name": "Jane Roe", "email": "jroe@globex.example", "company": "Globex"}]

def _search(query: str, company: str | None = None, limit: int = 5) -> str:
    q = query.lower()
    hits = [c for c in CONTACTS if q in c["name"].lower() or q in c["email"]]
    if company:
        hits = [c for c in hits if c["company"].lower() == company.lower()]
    return json.dumps(hits[:limit])

# Variant A: the "before" description. Same name, same code as variant B.
@tool(name="search_contacts")
def search_before():
    async def execute(query: str, company: str | None = None, limit: int = 5):
        """Searches contacts.

        Args:
            query: The query.
            company: Company.
            limit: Limit.
        """
        return _search(query, company, limit)
    return execute

# Variant B: the "after" description, rewritten from reading transcripts.
@tool(name="search_contacts")
def search_after():
    async def execute(query: str, company: str | None = None, limit: int = 5):
        """Find people in the CRM by partial name or email. Use this first whenever
        the user names a person but gives no email address. Returns up to `limit`
        matches, each with name, email and company. If several people match,
        narrow with `company` instead of guessing.

        Args:
            query: Part of a person's name or email, e.g. "jane" or "jane@".
            company: Optional company name to narrow results, e.g. "Acme".
            limit: Maximum matches to return (1 to 20). Default 5.
        """
        return _search(query, company, limit)
    return execute

@tool
def send_email():
    async def execute(to: str, subject: str, body: str):
        """Send an email.

        Args:
            to: Recipient email address.
            subject: Subject line.
            body: Message body.
        """
        return "queued (mock)"
    return execute

DATASET = [
    Sample(input="Email Jane from Acme that the demo moved to Friday.",
           target="jane@acme.example"),
    Sample(input="Let Jane Roe at Globex know the invoice is paid.",
           target="jroe@globex.example"),
    # ... 30 to 100 realistic tasks, split into a dev set and a held-out set
]

def tool_calls(state: TaskState):
    return [c for m in state.messages if isinstance(m, ChatMessageAssistant)
            for c in (m.tool_calls or [])]

@scorer(metrics=[accuracy(), stderr()])
def emailed_right_person():
    async def score(state: TaskState, target: Target):
        sent = [c.arguments.get("to") for c in tool_calls(state) if c.function == "send_email"]
        ok = sent == [target.text]              # exactly one email, to the right address
        return Score(value=1 if ok else 0, answer=str(sent))
    return score

@scorer(metrics=[mean()])
def num_tool_calls():
    async def score(state: TaskState, target: Target):
        return Score(value=len(tool_calls(state)))
    return score

@task
def contacts_ab(variant: str = "before"):
    search = search_before() if variant == "before" else search_after()
    return Task(
        dataset=DATASET,
        solver=[system_message("You are an assistant with CRM and email tools."),
                use_tools(search, send_email()),
                generate()],                    # generate() runs the tool loop
        scorer=[emailed_right_person(), num_tool_calls()],
        metadata={"variant": variant},
    )

Three details make this a fair test. The tool name is identical in both arms, so you are testing the description and not the name. The code is shared, so any difference is caused by the words. And the scorer checks the outcome (exactly one email, to the right person) while a second scorer records efficiency (tool calls per task). To finish the analysis, load both logs with read_eval_log, pair the per-sample scores by sample id, and bootstrap the difference as in Module 04.

MCP security testing

Everything a server advertises (tool names, descriptions, schemas, and every result it returns) lands in the model's context, and the model tends to treat it as trustworthy. That makes each installed server a supply-chain dependency that can talk directly to your model. Security researchers at Invariant Labs popularized the core attack names in April 2025, and the MCP specification now carries a security best practices page. This section gives you the attack shapes and a test for each; Module 25 covers red-teaming methodology, injection benchmarks and attack success metrics in depth.

The five attack shapes, and how to test each

• Tool poisoning. Instructions hidden in a tool description (or parameter descriptions, or results) that the user never sees but the model follows. Test: build a small suite of poisoned test servers with placeholder payloads, plant canary secrets (unique fake strings in files and environment variables), run your normal tasks, and measure the attack success rate (share of runs where a canary appears in any outbound call). Also scan descriptions statically for instruction-like text addressed to the model.
• Rug pull. A server behaves well when you review and approve it, then changes its tool definitions later (for example, after an update, or on the Nth session). Test: pin a hash of every approved manifest (the manifest test above, run on the consumer side), re-list tools every session and after every list_changed notification, and fail or re-prompt for approval on any diff. Your eval: a test server that swaps its description after the first session, and a check that the host notices.
• Over-permissioned servers. A server that can do far more than the use case needs (a "read tickets" integration holding a token that can delete repos). Test: enumerate every tool and credential scope; run adversarial tasks that ask the agent to do out-of-scope things and verify the server refuses, not just the model. Check that tools with side effects carry destructiveHint, but remember annotations are hints written by the server author; a malicious server can lie in them, so hosts must not treat them as a security boundary.
• Confused deputy. A server acting with its own broad authority on behalf of a caller who should not have it. The MCP security guidance describes this for servers that proxy to third-party APIs with a single static OAuth client, and warns against token passthrough (accepting tokens not issued for the server). Test: as user A, request user B's resource through the server; replay a token minted for a different audience; check that consent is per user and per client, and that the server validates token audience.
• Cross-server shadowing. One server's description alters how the model uses another server's tools (figure above). Test: install a canary "shadow" server alongside your trusted servers and compare the trusted tools' call arguments with and without it. Any difference in calls to trusted tools, when the task never needed the shadow server, is a finding.

Two public benchmarks give you ready-made attacks to start from: MCPSecBench, a systematic security benchmark and playground for MCP, and MCP-SafetyBench (ICLR 2026), which evaluates model safety with real-world MCP servers. Research on misleading descriptions ("Don't believe everything you read", 2026) measures how agents behave when a tool's description does not match what the tool actually does, which is a useful framing for your own tests: what happens when the text lies?

Putting it together: a tool eval plan

Here is how the pieces fit when you own an agent product that uses a mix of your own and third-party MCP servers.

Check yourself

Sources and further reading

• Anthropic Engineering, Writing effective tools for agents, with agents (September 2025). The loop, metrics, held-out evaluation and response-design advice in this module.
• Anthropic Engineering, Introducing advanced tool use on the Claude Developer Platform (tool search tool, deferred loading) and Code execution with MCP: building more efficient agents (November 2025).
• Model Context Protocol: specification (server primitives), Security best practices, MCP Inspector, and the MCP blog post Tool annotations as risk vocabulary: what hints can and can't do (March 2026).
• Luo et al. (Salesforce AI Research), MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers (2025); code.
• Wang et al. (Accenture), MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers (2025, ICLR 2026); code.
• LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on Challenging Queries (2025).
• Li et al., The Tool Decathlon (Toolathlon): Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution (2025, ICLR 2026).
• MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use (2025, ICLR 2026); code.
• Scale AI, MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers (2026) and the MCP Atlas leaderboard.
• LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools? (2025).
• RAG-MCP: Mitigating Prompt Bloat in LLM Tool Selection via Retrieval-Augmented Generation (2025) and Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models (ToolRet, ACL Findings 2025).
• Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions (2026) and From Docs to Descriptions: Smell-Aware Evaluation of MCP Server Descriptions (2026).
• Gorilla team, Berkeley Function Calling Leaderboard V4 and its format sensitivity post.
• Invariant Labs, MCP Security Notification: Tool Poisoning Attacks and WhatsApp MCP Exploited (2025); mcp-injection-experiments. Simon Willison, Model Context Protocol has prompt injection security problems (2025).
• MCPSecBench (2025), MCP-SafetyBench (2025, ICLR 2026), and Don't believe everything you read: Understanding and Measuring MCP Behavior under Misleading Tool Descriptions (2026).
• UK AISI and Meridian Labs, Inspect: Model Context Protocol tools. Inspect code in this module checked against inspect_ai 0.3.x.