Tool use, RAG and conversational agents

Most agents in production are text agents that call functions, look things up, and talk to people. This module shows exactly where they break, and how to grade them at every level: one function call, a chain of calls, a whole task that changes a database, a retrieval pipeline, and a full conversation with a user.

How function calling actually works

A language model can only produce text. It cannot check a database, send an email, or look up the weather. Function calling (also called tool use) is the trick that lets it act anyway: you describe some functions to the model, the model writes a structured request saying "please call this function with these arguments", and your code actually runs it and hands the result back.

Three pieces make this work:

• A tool schema. For each tool you give a name, a plain-English description, and a JSON Schema for its inputs. JSON Schema is a standard way to say "this argument is a string, this one is an integer between 1 and 10, this one is required". The description is not decoration: it is the only thing the model reads to decide when to use the tool.
• A tool call. When the model decides it needs a tool, it stops writing prose and emits a structured block. In Anthropic's Messages API this is a tool_use content block with an id, a name and an input object; OpenAI and Google use similar shapes with different field names.
• A tool result. Your code (the harness) validates the input, runs the real function, and appends a tool_result (linked by the call's id) to the conversation. If the tool failed, you still return a result, marked as an error, so the model can recover.

Here is the smallest possible example with the Anthropic Python SDK, so you can see the real shapes. You will grade exactly these objects later.

import anthropic

client = anthropic.Anthropic()

tools = [{
    "name": "get_order",
    "description": "Look up one order by its ID. Use this before answering any "
                   "question about an order's status, items or refund eligibility.",
    "input_schema": {
        "type": "object",
        "properties": {"order_id": {"type": "string", "description": "e.g. 'A17'"}},
        "required": ["order_id"],
    },
}]

resp = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=16000,
    tools=tools,
    messages=[{"role": "user", "content": "Where is my order A17?"}],
)

for block in resp.content:
    if block.type == "tool_use":            # step ③ in the diagram
        print(block.id, block.name, block.input)   # e.g. toolu_01..., get_order, {'order_id': 'A17'}
# resp.stop_reason == "tool_use" tells the harness it must run the tool and continue

What can go wrong, step by step

The diagram above has five decision points. Each has its own family of failures. Learning these names matters, because a good eval set has test cases aimed at each one, and a good error analysis (Module 12) sorts failures into these buckets.

A few of these deserve a closer look, because beginners tend to forget them:

• Calling when not needed is as real a failure as not calling. If a user asks "what's 2 + 2?" and the agent calls a web search, that costs money and latency. If a user asks something no tool can answer and the agent calls the closest-looking tool anyway, that is a hallucinated action. Benchmarks call the test for this irrelevance detection: give the model tools that do not fit the request and check that it declines to call any.
• Hallucinated tool names and arguments: the model calls cancel_booking when only cancel_reservation exists, or passes priority="urgent" to a function that has no priority parameter. Strict schema validation in the harness catches some of this at runtime (Anthropic's API offers strict: true on tool definitions to guarantee schema-valid inputs), but your eval should still count how often the model tries.
• Invented values are the most dangerous argument error because they pass type checks. order_id="A18" is a perfectly valid string; it is just the wrong order. Only a check against the reference value, or against the end state, catches it.
• Ignoring the result: the tool says "order not found" and the agent cheerfully reports the delivery date anyway. This is a grounding failure, the same family as RAG faithfulness failures later in this module.
• Looping: the agent calls the same search with the same query eight times. Always cap steps in your harness and record "hit the step cap" as its own failure category.

Four levels of tool-use evaluation

You can grade tool use at four levels of zoom. Lower levels are cheap, precise and easy to automate, but less realistic. Higher levels measure what users actually experience, but they are expensive, noisy and harder to grade. Strong teams run all four: many cheap low-level tests in CI, and fewer expensive end-to-end runs before a release.

Level 1: grading a single call (BFCL and AST matching)

The Berkeley Function Calling Leaderboard (BFCL), from the UC Berkeley Gorilla team, is the standard public benchmark for this level. Its core idea is simple and worth copying for your own evals: don't compare the model's call as a string, compare its structure.

AST matching, explained

An AST (abstract syntax tree) is what a programming language's parser turns code into: instead of the characters get_weather(city="NYC"), you get a tree with a function-name node and one argument node per parameter. Comparing trees instead of strings means that argument order, spacing and quoting style no longer matter. BFCL then checks each node against a list of possible answers, because many values are equally correct ("NYC", "New York, NY").

BFCL's published rules (from its V1 blog post) are a good template for your own grader:

• Function name must match. (Dots in names are converted to underscores for models that can't handle them, then converted back.)
• All parameters marked required in the schema must be present, and any parameter that is not in the schema is flagged as hallucinated.
• Types are checked strictly. Booleans must be real booleans, not the string "true". For Python tests only, an int is accepted where a float is expected; a float is never accepted for an int.
• Strings are compared case-insensitively, with all whitespace and a subset of punctuation (,./-_*^) removed, against a list of acceptable values (so several date or location formats can pass).
• Lists must match in order; dictionaries are checked key by key, ignoring key order.

AST versus executable evaluation

BFCL has two ways to judge a single call. AST evaluation checks the structure without running anything, which is fast, free and deterministic, but it can reject a call that would have worked (for example, a valid city spelling the annotators didn't list). Executable evaluation actually runs the call against a real or mock API and compares the response, which credits any call that produces the right result but depends on the API being available and stable. The trade-off is the same one you meet everywhere in this course: checking the path versus checking the outcome (Module 05).

BFCL's categories

Build it yourself: a pytest tool-call grader

Here is a self-contained grader in the BFCL style. It is deliberately a little looser than BFCL on numbers (it treats 5 and 5.0 as equal) and it returns a list of problems rather than just pass or fail, so failures are easy to categorize later. Save it as test_tool_calls.py and run pytest -q.

# test_tool_calls.py  -- run with: pytest -q
import re
import unicodedata
import pytest

JSON_TYPES = {"string": str, "integer": int, "number": (int, float),
              "boolean": bool, "array": list, "object": dict}

def type_ok(value, json_type: str) -> bool:
    # bool is a subclass of int in Python, so reject it explicitly for numbers
    if json_type in ("integer", "number") and isinstance(value, bool):
        return False
    return isinstance(value, JSON_TYPES[json_type])

def norm(v):
    """Normalize a value before comparing (similar to BFCL's string rules)."""
    if isinstance(v, str):
        v = unicodedata.normalize("NFKD", v).encode("ascii", "ignore").decode()
        v = re.sub(r"[,./\-_*^]", "", v.lower())   # drop some punctuation
        return re.sub(r"\s+", "", v)               # drop all whitespace
    if isinstance(v, bool):
        return v
    if isinstance(v, (int, float)):
        return float(v)                            # looser than BFCL: 5 == 5.0
    if isinstance(v, list):
        return [norm(x) for x in v]                # order matters
    if isinstance(v, dict):
        return {k: norm(x) for k, x in v.items()}  # key order does not
    return v

def grade_call(actual, expected, schema) -> list[str]:
    """Return a list of problems. Empty list = the call matches.
    actual:   {"name": ..., "args": {...}} or None if the model made no call
    expected: {"name": ..., "args": {param: [acceptable values]}} or None
              ("" in the acceptable list means the param may be omitted)"""
    if expected is None:                       # irrelevance case: must NOT call
        return [] if actual is None else [f"called {actual['name']} but no call was needed"]
    if actual is None:
        return ["no tool call, but one was needed"]
    if actual["name"] != expected["name"]:
        return [f"wrong tool: {actual['name']} (wanted {expected['name']})"]

    problems, args = [], actual.get("args", {})
    props, required = schema["properties"], set(schema.get("required", []))
    for p in required - args.keys():
        problems.append(f"missing required arg: {p}")
    for p, v in args.items():
        if p not in props:
            problems.append(f"hallucinated arg: {p}")
            continue
        if not type_ok(v, props[p]["type"]):
            problems.append(f"{p}: wrong type {type(v).__name__}, schema says {props[p]['type']}")
        allowed = expected["args"].get(p)
        if allowed is not None and norm(v) not in [norm(a) for a in allowed if a != ""]:
            problems.append(f"{p}: value {v!r} not in {allowed!r}")
    for p, allowed in expected["args"].items():
        if p not in args and p not in required and "" not in allowed:
            problems.append(f"missing arg the task needs: {p}")
    return problems

# ---------------------------------------------------------------- test data
SCHEMA = {"type": "object",
          "properties": {"city": {"type": "string"},
                         "unit": {"type": "string"},
                         "days": {"type": "integer"}},
          "required": ["city"]}
EXPECTED = {"name": "get_weather",
            "args": {"city": ["New York, NY", "NYC"], "unit": ["fahrenheit", "F", ""]}}

CASES = [
    ("exact",          {"name": "get_weather", "args": {"city": "New York, NY", "unit": "fahrenheit"}}, True),
    ("normalized",     {"name": "get_weather", "args": {"city": "new york ny"}},                         True),
    ("wrong tool",     {"name": "get_forecast", "args": {"city": "NYC"}},                                False),
    ("missing city",   {"name": "get_weather", "args": {"unit": "F"}},                                   False),
    ("hallucinated",   {"name": "get_weather", "args": {"city": "NYC", "priority": "high"}},             False),
    ("wrong type",     {"name": "get_weather", "args": {"city": "NYC", "days": "3"}},                    False),
    ("invented value", {"name": "get_weather", "args": {"city": "Newark"}},                              False),
]

@pytest.mark.parametrize("label,actual,should_pass", CASES, ids=[c[0] for c in CASES])
def test_grader(label, actual, should_pass):
    problems = grade_call(actual, EXPECTED, SCHEMA)
    assert (problems == []) == should_pass, problems

def test_irrelevance():
    # "Tell me a joke" with only get_weather available: the right move is no call
    assert grade_call(None, None, SCHEMA) == []
    assert grade_call({"name": "get_weather", "args": {"city": "NYC"}}, None, SCHEMA) != []

Level 2: multi-step chains

Real tasks need several calls where the output of one becomes the input of the next: look up the user, then their orders, then the one that matches, then refund it. Grading gets harder because there is usually more than one valid path. The agent might list all orders then filter, or search directly; it might re-read a record to double-check. An exact-sequence match would punish all of that.

BFCL V3 introduced multi-turn categories to tackle this. Per its V3 blog post there is a base multi-turn set (200 entries) plus augmented sets for missing parameters (the model should ask for the missing information rather than guess), missing functions (no available function can do it yet; the right move is to say so, and the function is provided in a later turn), and long context (lots of distracting data in results). The blog also describes a composite set combining all three. It grades with two checks:

• State-based: after the calls, compare the backend's state (files created, a watchlist changed) with the expected state.
• Response-based, subset-matched: the model's calls are correct if they contain the ground-truth calls as a subset, even with extra exploratory steps. This allows for exploring and recovering from errors, which a strict sequence match would reject.

For your own chains, a useful three-tier rubric is: (1) did the end state come out right; (2) were all essential calls made (a subset check, possibly with ordering constraints for steps that must precede others, like verify-before-refund); (3) were there any forbidden calls (a delete that should never happen, a write before authentication). Tier 3 is often the most important in production and the most often forgotten.

Level 3: whole tasks in stateful environments (τ-bench)

τ-bench (tau-bench, from Sierra, June 2024) is the benchmark that defined how to evaluate customer-service style agents. It combines three things that single-call benchmarks lack: a realistic policy, a stateful database, and a simulated user who has to be talked to.

The moving parts

• Domains. The original paper has retail (115 tasks) and airline (50 tasks). Each domain comes with a database (users, orders, flights), a set of Python tool functions over it, and a policy document, a plain-language rulebook like a real company's ("basic economy cannot be modified", "confirm with the user before any write action"). The policy goes into the agent's system prompt.
• The simulated user. Each task has a hidden instruction given only to a user-simulator LLM: who they are, what they want, what they know, and how they behave. The agent has to discover the goal through conversation, just like with a real customer.
• End-state grading. Each task is annotated with the gold actions (the write calls a correct agent should make). The grader replays them on a fresh copy of the database to get the goal state, then compares it with the database the agent actually left behind. It also checks that required outputs (say, a price the user asked for) appeared in the agent's replies. Both must pass for reward 1.

Why is end-state grading so powerful? Because it doesn't care how the agent got there. Read the order twice, apologize three times, search before lookup: all fine. But refund the wrong item, or refund when policy says no, and the database differs, and the task fails. It grades the thing the business actually cares about.

Build it yourself: end-state comparison

import copy, hashlib, json, re

def canonical(db: dict, ignore_keys=("updated_at", "request_id")) -> str:
    """Stable JSON for hashing: sorted keys, volatile fields removed."""
    def strip(x):
        if isinstance(x, dict):
            return {k: strip(v) for k, v in x.items() if k not in ignore_keys}
        if isinstance(x, list):
            return [strip(v) for v in x]
        return x
    return json.dumps(strip(db), sort_keys=True, separators=(",", ":"))

def db_hash(db: dict) -> str:
    return hashlib.sha256(canonical(db).encode()).hexdigest()

def replay(initial_db: dict, gold_actions: list[dict], tools: dict) -> dict:
    """Run the annotated gold write-actions on a fresh copy to get the goal state."""
    db = copy.deepcopy(initial_db)
    for a in gold_actions:                     # e.g. {"name": "refund_item", "args": {...}}
        tools[a["name"]](db, **a["args"])
    return db

def diff(a, b, path=""):
    """Yield every path where two nested structures differ (for debugging failures)."""
    if isinstance(a, dict) and isinstance(b, dict):
        for k in sorted(a.keys() | b.keys()):
            yield from diff(a.get(k), b.get(k), f"{path}.{k}")
    elif a != b:
        yield f"{path}: goal={a!r} agent={b!r}"

def squash(s: str) -> str:
    return re.sub(r"[\s,$]", "", s.lower())

def grade_episode(initial_db, gold_actions, tools, agent_final_db,
                  agent_messages: list[str], required_outputs: list[str]) -> dict:
    goal_db = replay(initial_db, gold_actions, tools)
    state_ok = db_hash(goal_db) == db_hash(agent_final_db)
    said = " ".join(squash(m) for m in agent_messages)
    outputs_ok = all(squash(o) in said for o in required_outputs)
    return {
        "reward": float(state_ok and outputs_ok),
        "state_ok": state_ok,
        "outputs_ok": outputs_ok,
        # the diff is what you read during error analysis:
        # it shows collateral damage (unintended writes) as well as missing ones
        "diff": list(diff(json.loads(canonical(goal_db)),
                          json.loads(canonical(agent_final_db))))[:20],
    }

pass^k: measuring reliability, not luck

τ-bench's second big contribution is a metric. You already know pass@k from coding benchmarks (Module 04): run k attempts and count the task solved if any attempt succeeds. That is the right metric when a human or a test suite picks the best attempt. It is the wrong metric for a customer-service agent, which gets one shot with each of thousands of customers who ask the same kind of thing.

τ-bench defines pass^k ("pass hat k"): the probability that all k independent attempts at a task succeed. With n trials per task and c successes, the unbiased estimator is:

pass^k = mean over tasks of C(c, k) / C(n, k)

where C(a, b) is "a choose b", the number of ways to pick b items from a. Compare pass@k = mean of 1 − C(n−c, k) / C(n, k). At k = 1 the two are identical. As k grows, pass@k rises towards 1 and pass^k falls: the gap between them is a direct picture of inconsistency. The τ-bench abstract reports that even gpt-4o succeeded on fewer than 50% of tasks and that its pass^8 in retail was below 25%.

from math import comb

def pass_hat_k(trials_per_task: list[list[bool]], k: int) -> float:
    """tau-bench pass^k: probability that k i.i.d. attempts at a task ALL succeed."""
    vals = []
    for trials in trials_per_task:
        n, c = len(trials), sum(trials)
        if n < k:
            raise ValueError(f"need at least k={k} trials per task, got {n}")
        vals.append(comb(c, k) / comb(n, k))   # comb(c, k) is 0 when c < k
    return sum(vals) / len(vals)

def pass_at_k(trials_per_task: list[list[bool]], k: int) -> float:
    """Codex pass@k: probability that at least one of k attempts succeeds."""
    vals = []
    for trials in trials_per_task:
        n, c = len(trials), sum(trials)
        vals.append(1 - comb(n - c, k) / comb(n, k))
    return sum(vals) / len(vals)

# toy example from the chart: 5 tasks x 8 trials
successes = [8, 7, 6, 4, 2]
results = [[True] * c + [False] * (8 - c) for c in successes]
for k in (1, 2, 4, 8):
    print(k, round(pass_at_k(results, k), 3), round(pass_hat_k(results, k), 3))
# 1 0.675 0.675
# 2 0.843 0.507
# 4 0.954 0.346
# 8 1.0   0.2

τ²-bench: when the user has hands too

τ²-bench (Sierra, June 2025) adds a telecom domain with dual control. In retail and airline, only the agent can change the world; the user just talks. In telecom, the user also has tools acting on their own device (turn airplane mode off, restart, check signal), and the agent has to guide them, like real tech support. The paper models this as a Dec-POMDP (a decentralized, partially observable decision process: two actors, each seeing only part of the state). This tests something single-control benchmarks cannot: whether the agent can coordinate and communicate, not just act.

τ²-bench also ships fixed versions of the airline and retail tasks and ablation modes that help diagnose failures. Running without a user (the agent gets the full task directly) or with an oracle plan separates reasoning failures from communication failures. The repository (checked September 2026) now also lists a banking_knowledge domain that adds retrieval over a knowledge base, voice evaluation, and a newer iteration branded τ³-bench. Leaderboard: taubench.com. The voice variant is covered in Module 09.

Other stateful benchmarks

• AppWorld (Stony Brook and AI2, ACL 2024): a simulated world of 9 everyday apps with 457 APIs and about 100 simulated people. The agent writes code that calls the APIs to finish 750 tasks. Grading uses state-based unit tests that also check for collateral damage: the task fails if the agent made unintended changes, like deleting the wrong song. It reports TGC (task goal completion) and SGC (scenario goal completion, where all task variants in a scenario must pass, a reliability measure in the spirit of pass^k).
• ToolBench / ToolLLM (Tsinghua, ICLR 2024): 16,464 real REST APIs from RapidAPI. Scored by pass rate and win rate, both judged by an LLM. Real APIs go offline and change, so results were hard to reproduce; StableToolBench replaced them with a cached and simulated virtual API server. It is a useful lesson: if your environment is not frozen, your eval is not repeatable.
• ACEBench (Huawei Noah's Ark and USTC, January 2025): three groups: Normal (atomic, single and multi-turn, similar APIs, user preferences), Special (imperfect instructions with missing or wrong parameters, where the right move is to ask or flag), and Agent (a sandbox with a simulated user). It grades with AST-style checks and end-state checks and deliberately uses no LLM judge.

Level 4: general assistants and search agents

At the top of the staircase, the agent has open-ended tools (a browser, a code interpreter, file readers) and the task is a question, not a database change. The grading trick here is to pick questions whose answers are short and verifiable, even though finding them is hard.

• GAIA (Meta, Hugging Face and AutoGPT, November 2023): 466 real-world questions needing browsing, file handling, multimodality and reasoning, in three difficulty levels (300 in the held-out test set). Answers are a number, a few words or a list, graded by quasi-exact match. At release, humans scored 92% and GPT-4 with plugins about 15%. Caveat: the validation answers are public, so treat validation scores as possibly contaminated.
• BrowseComp (OpenAI, April 2025): 1,266 questions that are "hard to find but easy to verify", built by starting from a fact and writing a question with many intertwined constraints. At launch, OpenAI's Deep Research scored 51.5%, GPT-4o with browsing 1.9%, and human trainers solved 29.2% within two hours. It is the hard end of agentic search. BrowseComp-Plus fixes the document corpus so that results are reproducible and retrieval can be studied separately.

These are also where agentic RAG lives: instead of retrieving once and answering, a search agent decides what to search, reads, reformulates and searches again. Evaluating it combines everything in this module: tool-call quality (were the queries sensible?), retrieval quality (did it find the evidence?), faithfulness (does the answer stick to what it found?), and end-task accuracy. Web agents are covered further in Module 08.

Evaluating the tools themselves

So far we held the tools fixed and graded the agent. But for many teams the biggest lever is the tools: their names, descriptions, parameters and what they return. Anthropic's engineering post "Writing effective tools for agents, with agents" (September 2025) describes an eval-driven loop for this.

The key points from the post:

• Tasks should be realistic and need several calls. A strong example: "Schedule a meeting with Jane next week to discuss our latest Acme Corp project. Attach the notes from our last project planning meeting and reserve a conference room." A weak one: "Schedule a meeting with jane@acme.corp next week." The weak task hands the agent the exact argument and tests almost nothing.
• Collect more than accuracy: runtime of each tool call and of the whole task, total number of tool calls, token consumption, and tool error frequency. Many redundant calls suggest a tool should paginate or filter; many errors suggest unclear parameters.
• Read the transcripts, including the agent's reasoning. The post notes that "what agents omit in their feedback and responses can often be more important than what they include."
• Let Claude help: concatenate transcripts and give them to Claude Code to refactor the tools, then re-run on held-out tasks.

The post's design advice doubles as a checklist of things to measure: namespacing related tools (e.g. asana_projects_search) to reduce confusion between similar tools; returning meaningful context (names rather than opaque UUIDs); a response_format option so the agent can ask for concise or detailed output (the post reports concise responses using about a third of the tokens); and token efficiency through pagination, filtering and truncation with sensible defaults.

Evaluating RAG

RAG (retrieval-augmented generation) means: before answering, fetch relevant documents and put them in the prompt. It is the most common "tool" of all, and it has its own mature evaluation vocabulary. The single most important idea: a RAG system has two parts that fail differently, so measure them separately. If the final answer is wrong, you need to know whether the retriever never found the right document, or the generator had it and ignored it.

Retrieval metrics, by hand

These are classic information-retrieval metrics. They need relevance labels: for each test query, which documents or chunks are relevant. Let k be how many results you keep (the "top k").

• Recall@k: of all relevant chunks, what fraction appear in the top k? "Did we find the evidence?" This is usually the most important retrieval metric for RAG, because the generator can't use what it never sees.
• Precision@k: of the top k, what fraction are relevant? "How much noise did we add?" Irrelevant chunks cost tokens and can distract the generator.
• MRR (mean reciprocal rank): for each query take 1 / (rank of the first relevant result), then average over queries. It rewards getting one good result near the top.
• nDCG@k (normalized discounted cumulative gain): each relevant result earns credit discounted by position (1 / log2(rank + 1)), summed (DCG), then divided by the best possible DCG (IDCG), giving 0 to 1. It rewards putting relevant results high and supports graded relevance (2 = perfect, 1 = partial).

import math

def recall_at_k(ranked_ids, relevant, k):     return len(set(ranked_ids[:k]) & relevant) / len(relevant)
def precision_at_k(ranked_ids, relevant, k):  return len(set(ranked_ids[:k]) & relevant) / k

def reciprocal_rank(ranked_ids, relevant):
    for i, doc in enumerate(ranked_ids, start=1):
        if doc in relevant:
            return 1 / i
    return 0.0

def ndcg_at_k(ranked_ids, gains: dict, k):
    """gains: doc_id -> graded relevance (e.g. 0, 1, 2). Missing = 0."""
    dcg = sum(gains.get(d, 0) / math.log2(i + 1) for i, d in enumerate(ranked_ids[:k], start=1))
    ideal = sorted(gains.values(), reverse=True)[:k]
    idcg = sum(g / math.log2(i + 1) for i, g in enumerate(ideal, start=1))
    return dcg / idcg if idcg else 0.0

ranked = ["gift", "window30", "shipping", "receipt", "hours"]
relevant = {"window30", "receipt", "original_card"}
print(recall_at_k(ranked, relevant, 5), precision_at_k(ranked, relevant, 5),
      reciprocal_rank(ranked, relevant), round(ndcg_at_k(ranked, {d: 1 for d in relevant}, 5), 2))
# 0.666... 0.4 0.5 0.5      MRR = mean of reciprocal_rank over all test queries

Generation metrics: faithfulness and friends

Once the chunks are in the prompt, the generator can still fail in two main ways: it can say things the chunks don't support (a hallucination relative to the context), or it can answer a different question from the one asked.

• Faithfulness (also called groundedness): is every claim in the answer supported by the retrieved context? Ragas defines it as number of claims supported by the context / total claims in the response. Note that this is about the context, not about the world: an answer can be faithful and still wrong if the retrieved document was wrong.
• Answer relevancy: does the answer address the question? Ragas computes it by having an LLM generate several questions (3 by default) that the answer would be a good answer to, then taking the mean cosine similarity between their embeddings and the original question's embedding.
• Context precision (Ragas): are the relevant chunks ranked near the top? The mean of precision@k at each relevant position, as in the worked example above. Relevance is judged by an LLM against a reference answer, so you don't need hand labels per chunk.
• Context recall (Ragas): number of claims in the reference answer supported by the retrieved context / total claims in the reference. A label-free cousin of recall@k: it needs a reference answer instead of chunk labels.

With Ragas (current API)

Ragas is the most widely used open-source library for these metrics. Its current docs use a "collections" API in ragas.metrics.collections; the older SingleTurnSample style is marked for deprecation. Each metric takes an LLM (and, for answer relevancy, an embedding model) and exposes async ascore() and sync score(). The docs' examples use an OpenAI client; check the Ragas docs for other providers.

import asyncio
from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.embeddings.base import embedding_factory
from ragas.metrics.collections import (
    Faithfulness, AnswerRelevancy, ContextPrecision, ContextRecall,
)

client = AsyncOpenAI()
llm = llm_factory("gpt-4o-mini", client=client)
emb = embedding_factory("openai", model="text-embedding-3-small", client=client)

sample = dict(
    user_input="Can I get a refund on a gift card?",
    response="Yes, gift cards can be refunded within 14 days.",
    retrieved_contexts=["Refunds are accepted within 30 days of delivery with a receipt. "
                        "Gift cards are non-refundable."],
    reference="No. Gift cards are non-refundable.",
)

async def main():
    faith = await Faithfulness(llm=llm).ascore(
        user_input=sample["user_input"], response=sample["response"],
        retrieved_contexts=sample["retrieved_contexts"])
    relev = await AnswerRelevancy(llm=llm, embeddings=emb).ascore(
        user_input=sample["user_input"], response=sample["response"])
    cprec = await ContextPrecision(llm=llm).ascore(
        user_input=sample["user_input"], reference=sample["reference"],
        retrieved_contexts=sample["retrieved_contexts"])
    crec = await ContextRecall(llm=llm).ascore(
        user_input=sample["user_input"], reference=sample["reference"],
        retrieved_contexts=sample["retrieved_contexts"])
    print(f"faithfulness={faith.value:.2f} relevancy={relev.value:.2f} "
          f"ctx_precision={cprec.value:.2f} ctx_recall={crec.value:.2f}")
    # Expect: faithfulness low (the answer contradicts the context), context recall high
    # (the context DID contain the reference fact). Diagnosis: a generator problem, not retrieval.

asyncio.run(main())

From scratch: a faithfulness judge you fully control

Libraries are convenient, but every metric is "just" a prompt plus arithmetic, and writing your own makes the judge's behavior visible and tunable. This version uses the Anthropic SDK with structured output so the verdicts always parse. Validate any judge like this against human labels before trusting it (Module 13).

import json
import anthropic

client = anthropic.Anthropic()
JUDGE_MODEL = "claude-sonnet-5-5"     # a cheaper model is common for judges; validate it first

SCHEMA = {
    "type": "object",
    "properties": {
        "claims": {"type": "array", "items": {
            "type": "object",
            "properties": {
                "claim":    {"type": "string"},
                "verdict":  {"type": "string", "enum": ["supported", "contradicted", "not_found"]},
                "evidence": {"type": "string"},   # quote from the context, or ""
            },
            "required": ["claim", "verdict", "evidence"],
            "additionalProperties": False,
        }},
    },
    "required": ["claims"],
    "additionalProperties": False,
}

PROMPT = """You are checking whether an answer is supported by retrieved context.
1. Split the ANSWER into atomic factual claims (one fact each). Skip greetings and hedges.
2. For each claim, decide using ONLY the CONTEXT, not your own knowledge:
   supported    = the context states or directly implies it (quote the evidence)
   contradicted = the context says otherwise (quote the evidence)
   not_found    = the context does not address it

{context}


{answer}
"""

def faithfulness(answer: str, contexts: list[str]) -> dict:
    resp = client.messages.create(
        model=JUDGE_MODEL,
        max_tokens=4000,
        messages=[{"role": "user", "content": PROMPT.format(
            context="\n---\n".join(contexts), answer=answer)}],
        output_config={"format": {"type": "json_schema", "schema": SCHEMA}},
    )
    claims = json.loads(next(b.text for b in resp.content if b.type == "text"))["claims"]
    supported = sum(c["verdict"] == "supported" for c in claims)
    return {
        "score": supported / len(claims) if claims else 1.0,   # no claims: nothing to be unfaithful about
        "unsupported": [c for c in claims if c["verdict"] != "supported"],  # read these!
    }

print(faithfulness("You can get a refund within 30 days if you have a receipt, "
                   "and gift cards can be refunded within 14 days.",
                   ["Refunds are accepted within 30 days of delivery with a receipt. "
                    "Gift cards are non-refundable."]))
# e.g. {'score': 0.67, 'unsupported': [{'claim': 'Gift cards can be refunded within 14 days', ...}]}

Evaluating conversational agents

A conversational agent is judged not by one reply but by a whole dialogue. That brings failure modes that no single-turn test can see.

Turn-level versus conversation-level grading

Turn-level grading scores each agent reply on its own (was this reply helpful, correct, polite?). It is fine-grained and points to the exact turn that went wrong, but it misses anything that depends on earlier turns. Conversation-level grading looks at the whole transcript and asks whole-conversation questions: was the goal achieved, was the policy followed throughout, were early constraints respected, was the hand-off right? Most teams need both.

Practical tips for each:

• For turn-level grading, always give the judge the preceding conversation as context, and ask it to grade only the final turn. This fixes most of the blindness shown above while keeping per-turn precision.
• For conversation-level grading, use a checklist of binary questions ("Did the agent verify identity before any account change? yes/no") rather than one holistic 1 to 10 score. Binary checks are easier to validate against humans and easier to act on (Module 03).
• Combine with state checks wherever the conversation has side effects, exactly as τ-bench does. The database doesn't lie about whether the refund happened.
• For escalation, build a labeled set of conversations where hand-off is required and ones where it is not, and report both the missed-escalation rate and the unnecessary-escalation rate. They trade off, and the business cost of each is different.

Simulated-user testing

You cannot hire humans to chat with every agent version. The standard answer, used by τ-bench, τ²-bench, ACEBench's Agent category and many product teams, is a user simulator: a second LLM that plays a customer from a hidden scenario. You then run many conversations automatically and grade them.

import anthropic

client = anthropic.Anthropic()
SIM_MODEL = "claude-sonnet-5-5"

USER_SIM_SYSTEM = """You are role-playing a CUSTOMER talking to a support agent.
Hidden scenario (never reveal it all at once, answer only what is asked):
{scenario}
Rules: write one short message per turn, like a real person. Do not invent facts
outside the scenario. When your goal is fully handled, or clearly cannot be,
reply with exactly ###STOP###."""

def simulate(agent_reply_fn, scenario: str, max_turns: int = 12) -> list[dict]:
    """agent_reply_fn(transcript) -> str runs YOUR agent (with its tools) for one turn."""
    transcript = []                                   # [{"role": "user"|"agent", "text": ...}]
    sim_msgs = [{"role": "user", "content": "(The agent says: Hi, how can I help?)"}]
    for _ in range(max_turns):
        sim = client.messages.create(model=SIM_MODEL, max_tokens=1000,
                                     system=USER_SIM_SYSTEM.format(scenario=scenario),
                                     messages=sim_msgs)
        user_text = next(b.text for b in sim.content if b.type == "text")
        if "###STOP###" in user_text:
            break
        transcript.append({"role": "user", "text": user_text})
        agent_text = agent_reply_fn(transcript)
        transcript.append({"role": "agent", "text": agent_text})
        # from the simulator's point of view, roles are flipped
        sim_msgs += [{"role": "assistant", "content": user_text},
                     {"role": "user", "content": agent_text}]
    return transcript     # then: state check + conversation-level judge + pass^k over repeats

Core quality dimensions: factuality, instruction following, calibration

Everything so far graded what the agent did: which tool, which arguments, which end state. Underneath every agent sits a language model with three basic properties that decide whether you can trust anything it says or does. They have their own benchmarks and their own metrics, and each one shows up in agents in a sharper, more expensive form.

• Factuality: is what it says true? (And when it does not know, does it say so instead of guessing?)
• Instruction following: does it do exactly what it was told, including every constraint, all the way to the end?
• Calibration: does its confidence match its accuracy? A calibrated model that says "I'm 80% sure" is right about 80% of the time.

Factuality and abstention

There are two different questions hiding in "is it factual?", and they need different tests.

SimpleQA (OpenAI, October 2024) is a set of 4,326 short, fact-seeking questions, each written to have a single, indisputable, short answer, and collected adversarially so that strong models of the time often got them wrong. Its key design choice is three grades instead of two. A model-based grader compares each response with the reference and labels it:

• correct: the answer matches the reference without contradicting it;
• incorrect: the answer contradicts the reference (a hallucination);
• not attempted: the model declined or hedged without committing ("I don't know").

With three grades you can separate "knows a lot" from "knows what it knows". The paper reports overall correct, correct given attempted (accuracy on the questions the model chose to answer), and an F-score, the harmonic mean of those two. A worked example makes the point.

Worked F-score for Model B: 2 × 0.38 × 0.76 / (0.38 + 0.76) = 0.5776 / 1.14 ≈ 0.51. This connects to a bigger argument. OpenAI's September 2025 essay "Why language models hallucinate" argues that most benchmarks grade right-or-wrong only, so guessing always scores at least as well as saying "I don't know", and models trained and selected on those scores learn to guess. Its proposed fix is on the eval side: penalize confident errors more than abstentions, for example by telling the model a confidence threshold t and taking off t / (1 − t) points for a wrong answer (at t = 0.75, a wrong answer costs 3 points and "I don't know" costs 0). When you build your own factuality eval, decide up front what an abstention is worth; if you do not, your eval rewards bluffing.

SimpleQA Verified (Google DeepMind and Google Research, September 2025) is a curated 1,000-prompt subset of SimpleQA, with a Kaggle leaderboard. The authors report that the original set has problems such as noisy or incorrect reference answers, topical skew and redundant questions, and filtered and re-checked it to fix them. The lesson is general: any factuality benchmark is only as good as its reference answers, and those go stale as the world changes (Module 18).

FACTS Grounding (Google DeepMind, announced December 2024, paper January 2025, Kaggle leaderboard) tests grounded factuality on long inputs. Each example is a document (up to roughly 32,000 tokens), a system instruction to answer only from it, and a user request such as a summary or a question. Grading has two phases: first, is the response eligible (does it actually address the request, rather than dodging with something short and safe)? Then, is every claim grounded in the document? Both are judged by several LLM judges from different model families, and the scores are averaged, which reduces the self-preference bias covered in Module 13. The eligibility phase is worth copying: without it, a model can score perfectly on "no unsupported claims" by saying almost nothing. DeepMind has since extended the FACTS name into a broader suite of factuality benchmarks, so check which variant a quoted score refers to.

TruthfulQA (Lin, Hilton and Evans, 2021) is the older classic: 817 questions across 38 categories, built so that a model imitating common human misconceptions ("what happens if you crack your knuckles a lot?") answers falsely. It is still widely quoted, so know its issues:

• It measures imitative falsehoods (repeating popular myths), not general factual accuracy, and not hallucination in the sense of inventing details.
• The original multiple-choice version turned out to be gameable: Alex Turner showed that simple heuristics that look only at the answer options, never at the question, score surprisingly well. In response, the authors published a new binary-choice version (January 2025) and recommend it over the old multiple-choice format.
• The generative version needs a judge (originally a fine-tuned "GPT-judge"), so scores depend on the grader.
• It is old and public, so contamination is likely for recent models.

Instruction following

"Follows instructions well" is vague, so the most useful benchmarks restrict themselves to instructions a program can check. IFEval (Zhou et al., Google, November 2023) has about 500 prompts, each carrying one or more verifiable instructions drawn from 25 types: "answer in fewer than N words", "do not use any commas", "include the keyword X at least 3 times", "wrap the whole response in JSON", "end with the exact phrase ...". Each type has a small Python checker, so there is no judge and no ambiguity.

The checkers really are this small, which is why you can copy the idea for your own product's rules:

import re

# Each checker takes the response text and returns True if the instruction was followed.
CHECKERS = {
    "max_words":   lambda text, n: len(text.split()) < n,
    "no_commas":   lambda text: "," not in text,
    "ends_with":   lambda text, phrase: text.strip().endswith(phrase),
    "keyword_min": lambda text, kw, k: len(re.findall(rf"\b{re.escape(kw)}\b", text, re.I)) >= k,
}

def grade(response: str, instructions: list[tuple]) -> dict:
    """instructions: [("max_words", 60), ("no_commas",), ("ends_with", "Any questions?")]"""
    results = [CHECKERS[name](response, *args) for name, *args in instructions]
    return {"instruction_level": sum(results) / len(results),   # fraction of instructions followed
            "prompt_level": float(all(results))}                  # 1.0 only if every one passed

resp = "Meet Aero: light, fast and quiet. Built for long days on the road. Any questions?"
print(grade(resp, [("max_words", 60), ("no_commas",), ("ends_with", "Any questions?")]))
# {'instruction_level': 0.666..., 'prompt_level': 0.0}

IFEval became a standard target, and that caused the familiar problem: models got very good at exactly those 25 constraint types. IFBench (Pyatkin et al., Ai2, July 2025, NeurIPS 2025 Datasets and Benchmarks) tests generalization instead, with 58 new, diverse constraints chosen to be out of distribution relative to IFEval. The paper reports that most models overfit to the small set of constraint types in existing benchmarks and generalize poorly to unseen ones, which is the signature of optimizing for a benchmark's specific format rather than the skill. The same paper also uses the constraint checkers as rewards for reinforcement learning, a pattern Module 13 returns to.

Calibration

A model is calibrated if its stated confidence matches how often it is right. You can get a confidence number three ways: ask for it ("give a confidence from 0 to 100", called verbalized confidence), read the probability the model assigned to its answer tokens (log-probs, when the API exposes them), or sample the same question several times and use the agreement rate (consistency). Then you check the numbers against reality with a reliability diagram: group answers into bins by confidence, and for each bin plot the actual accuracy. A perfectly calibrated model sits on the diagonal. Bars below it mean overconfidence.

The standard summary number is expected calibration error (ECE): the weighted average of |accuracy − confidence| across bins, with each bin weighted by its share of the answers. Here it is in code, reproducing the worked numbers above:

import numpy as np

def expected_calibration_error(conf, correct, n_bins=10):
    """conf: stated confidence per answer (0..1). correct: 1 if the answer was right, else 0."""
    conf, correct = np.asarray(conf, float), np.asarray(correct, float)
    edges = np.linspace(0.0, 1.0, n_bins + 1)
    idx = np.digitize(conf, edges[1:-1])          # bin index 0..n_bins-1 (1.0 lands in the top bin)
    ece = 0.0
    for b in range(n_bins):
        in_bin = idx == b
        if not in_bin.any():
            continue
        gap = abs(correct[in_bin].mean() - conf[in_bin].mean())
        ece += in_bin.mean() * gap                # weight = share of all answers in this bin
    return ece

# The worked example: (confidence, answers in bin, how many were right)
bins = [(0.55, 10, 5), (0.65, 10, 6), (0.75, 20, 13), (0.85, 20, 14), (0.95, 40, 30)]
conf, correct = [], []
for c, n, k in bins:
    conf += [c] * n
    correct += [1] * k + [0] * (n - k)

print(f"accuracy={np.mean(correct):.2f} mean_conf={np.mean(conf):.2f} "
      f"ECE={expected_calibration_error(conf, correct):.3f}")
# accuracy=0.68 mean_conf=0.82 ECE=0.140

How each dimension shows up in agents

In agents, all three properties turn into specific, testable behaviors. The most important one has a name: the false success claim, where the agent's final message says the task is done when the environment says otherwise. It is a factuality failure (the claim is false) and a calibration failure (the agent was sure without checking) at once, and users find it more damaging than an honest "I couldn't finish".

Worked example for claimed-success precision: across 100 runs, the agent says "done" in 80. The verifier confirms 60 of those. Claimed-success precision is 60 / 80 = 75%, so one "done" in four is false. Suppose a new prompt makes the agent verify before claiming: it now says "done" in 66 runs and 62 are real (94% precision), while the true success rate barely moves (60 to 62). Measured by success rate alone, that change looks like noise; measured by the thing users experience, it is a large win.

The benchmark map

A summary of the public benchmarks in this module. Sizes and dates are from the original papers or official pages; for current leaderboard numbers, go to the leaderboard itself, since they move monthly.

Putting it together: an eval plan for a support agent

Suppose you are shipping a support agent with 12 tools, a 6-page refund policy, and a knowledge base. A sensible, layered plan:

1. Tool-call unit tests (Level 1): 150 to 300 single-turn requests with expected calls, including 20% or more where the right answer is no call or ask a clarifying question. Graded by the pytest grader above. Runs on every commit, takes minutes.
2. Retrieval tests: 100 or more questions with labeled relevant chunks. Track recall@5 and nDCG@5 whenever you change chunking, embeddings or the index.
3. Faithfulness monitor: the faithfulness judge on RAG answers, both offline and on a sample of production traffic (Module 15).
4. Stateful scenarios (Level 3): 50 to 100 τ-bench-style tasks with a sandbox database, hidden-scenario user simulator, end-state grading, and a conversation-level checklist (policy, constraints, escalation). Run 4 to 8 trials each and report pass^1 and pass^k with confidence intervals before every release.
5. Error analysis loop: read failing transcripts weekly, tag them with the failure map from the start of this module, and turn each new failure type into new test cases (Module 12).

Check yourself

Sources and further reading

• Gorilla team, UC Berkeley, Berkeley Function Calling Leaderboard (V1 blog: AST rules, categories), BFCL V3: multi-turn and multi-step, and the live leaderboard.
• Yao, Shinn, Razavi, Narasimhan (Sierra), τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024).
• Barres, Dong, Ray, Si, Narasimhan (Sierra), τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (2025); tau2-bench repository; taubench.com leaderboard.
• Trivedi et al., AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents (ACL 2024).
• Qin et al., ToolLLM: Facilitating LLMs to Master 16000+ Real-world APIs (ICLR 2024); Guo et al., StableToolBench (2024).
• Chen et al., ACEBench: Who Wins the Match Point in Tool Usage? (2025).
• Mialon et al., GAIA: a benchmark for General AI Assistants (2023).
• Wei et al. (OpenAI), BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents (2025).
• Anthropic, Writing effective tools for agents, with agents (September 2025).
• Anthropic docs, Tool use with Claude.
• Ragas docs: Faithfulness, Answer relevancy, Context precision, Context recall.
• Wei et al. (OpenAI), Introducing SimpleQA (October 2024; paper arXiv:2411.04368).
• Google DeepMind and Google Research, SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge (September 2025); Kaggle leaderboard.
• Jacovi et al. (Google DeepMind), The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input (January 2025); DeepMind blog; Kaggle leaderboard.
• Lin, Hilton, Evans, TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021); Evans, Chua, Lin, New, improved multiple-choice TruthfulQA; Turner, Gaming TruthfulQA: simple heuristics exposed dataset weaknesses.
• Zhou et al. (Google), Instruction-Following Evaluation for Large Language Models (IFEval, November 2023).
• Pyatkin et al. (Ai2), Generalizing Verifiable Instruction Following (IFBench, July 2025).
• OpenAI, Why language models hallucinate (September 2025).
• Guo, Pleiss, Sun, Weinberger, On Calibration of Modern Neural Networks (2017): reliability diagrams and ECE.
• Chen et al. (OpenAI), Evaluating Large Language Models Trained on Code (the pass@k estimator).