Voice agents

A voice agent is a text agent that has to listen through a noisy phone line, decide in a fraction of a second whether you have finished talking, and answer out loud before the silence gets awkward. Every one of those extra jobs is a new way to fail, and a new thing to measure.

Why voice is its own problem

In Module 06 you learned to evaluate a conversational agent that reads and writes text: did it call the right tool, follow the policy, reach the goal? A voice agent has to do all of that plus a set of jobs that do not exist in text at all.

Five things make voice different from text, and each one shows up later as a metric:

Two terms first, because they appear everywhere. ASR (automatic speech recognition), also called STT (speech to text), turns audio into text. TTS (text to speech) turns text into audio. A VAD (voice activity detector) is a small, fast model that answers one question many times a second: "is someone speaking right now?"

Two architectures, two kinds of measurability

There are two broad ways to build a voice agent, and which one you have decides what you can measure.

Cascaded: VAD → ASR → LLM → TTS

The cascaded (or pipeline) design chains separate models. The VAD decides when the caller stopped talking. The ASR model produces a transcript. A normal text LLM, with tools, writes a reply. A TTS model speaks it. Frameworks such as Pipecat and LiveKit Agents are built around this shape, and let you swap each piece independently.

• Strength for evals: every arrow in the diagram is text or a timestamp you can log. You can compute WER on the ASR output, replay the same transcript through a new LLM without any audio, and time each stage separately. Debugging is a matter of finding which box broke.
• Weakness: information is thrown away at each boundary. The LLM never hears that the caller sounded angry or hesitant, only the words. Latencies add up across stages. And the ASR's mistakes become "facts" to the LLM.

Speech-to-speech: one native audio model

A speech-to-speech (S2S) or native audio model takes audio in and produces audio out directly, with the reasoning happening inside one network. OpenAI's Realtime API and Google's Gemini Live API (with its native audio models) are the best-known hosted examples. They usually still emit a text transcript of what they said, and they still call tools, but there is no separate ASR stage you can grade on its own.

• Strength: the model can hear tone, hesitation and emphasis, and can respond with appropriate prosody. There are fewer hops, so latency can be lower.
• Weakness for evals: it is a black box between audio in and audio out. You cannot compute "the ASR's WER" because there is no ASR. You grade end-to-end behavior. Transcripts the API emits are the model's own rendering, not ground truth, so treat them as a convenience, not as evidence.

Half-duplex vs full-duplex turn-taking

A second design axis is how the agent shares the line. Half-duplex means only one party talks at a time, like a walkie-talkie: the agent waits for the caller to finish, speaks, and may not listen while speaking. Full-duplex means the agent listens while it speaks, like a human on the phone: it can hear "mm-hm", notice an interruption, and stop mid-sentence. Most production cascades are "half-duplex with barge-in": they do not produce overlapping speech on purpose, but they keep a VAD running during playback so the caller can interrupt. True full-duplex models (which can also backchannel and overlap deliberately) are an active research area, which is why benchmarks such as Full-Duplex-Bench exist.

The layered eval: component, conversation, system

A voice call can fail at any of three altitudes, and you need metrics at each one. Think of it as the same testing pyramid you know from software, reshaped for calls.

The rest of this module walks up the stack, then covers how to generate test calls (simulation), how to grade audio, the public benchmarks, robustness, and safety.

Component level: the ear (ASR and WER)

Word error rate, from intuition to formula

Word error rate (WER) is the standard ASR metric. The intuition: line up what was actually said (the reference, usually a human transcript) with what the ASR produced (the hypothesis), and count the minimum number of word edits needed to turn one into the other. There are three kinds of edit:

• Substitution (S): a word was heard as a different word ("a" became "the").
• Deletion (D): a spoken word is missing from the transcript.
• Insertion (I): the transcript has a word nobody said.

Then WER = (S + D + I) / N, where N is the number of words in the reference. The "minimum number of edits" is the Levenshtein edit distance, computed on words instead of characters. (Character error rate, CER, is the same idea on characters, and is preferred for languages without spaces between words, such as Chinese or Japanese.)

Walk through it slowly. The reference has 8 words. Six words line up perfectly ("hits"). "a" was heard as "the" (1 substitution). "two" vanished (1 deletion). "thirty" appeared from nowhere (1 insertion). Three edits over eight reference words is 0.375, or 37.5%. There is no cheaper alignment: you could try to match "thirty" against "two", but then you would still need to account for "at seven" being in the wrong place, which costs more edits.

WER in code: the algorithm

Here is the whole algorithm in plain Python. It fills a dynamic-programming table (the same one used for spelling correction), then walks backwards through it to count which kind of edit was used at each step.

def wer_details(reference: str, hypothesis: str):
    """Word error rate via Levenshtein distance on word lists.
    Returns (wer, substitutions, deletions, insertions, hits)."""
    ref = reference.lower().split()
    hyp = hypothesis.lower().split()
    R, H = len(ref), len(hyp)
    # d[i][j] = minimum edits to turn ref[:i] into hyp[:j]
    d = [[0] * (H + 1) for _ in range(R + 1)]
    for i in range(R + 1):
        d[i][0] = i            # delete every reference word
    for j in range(H + 1):
        d[0][j] = j            # insert every hypothesis word
    for i in range(1, R + 1):
        for j in range(1, H + 1):
            cost = 0 if ref[i - 1] == hyp[j - 1] else 1
            d[i][j] = min(d[i - 1][j - 1] + cost,   # match or substitution
                          d[i - 1][j] + 1,          # deletion
                          d[i][j - 1] + 1)          # insertion
    # Walk back through the table to count each kind of edit
    i, j, S, D, I, hits = R, H, 0, 0, 0, 0
    while i > 0 or j > 0:
        if i > 0 and j > 0 and d[i][j] == d[i - 1][j - 1] + (ref[i - 1] != hyp[j - 1]):
            if ref[i - 1] == hyp[j - 1]:
                hits += 1
            else:
                S += 1
            i, j = i - 1, j - 1
        elif i > 0 and d[i][j] == d[i - 1][j] + 1:
            D += 1
            i -= 1
        else:
            I += 1
            j -= 1
    return (S + D + I) / max(R, 1), S, D, I, hits

ref = "please book a table for two at seven"
hyp = "please book the table for at seven thirty"
print(wer_details(ref, hyp))   # (0.375, 1, 1, 1, 6)

WER in code: the library way, with normalization

In practice you use jiwer, a small, fast Python library. The important part is not the call itself but the text normalization you apply first. Without it, "Book a table." and "book a table" differ by capitalization and punctuation, and your WER measures formatting, not hearing.

import jiwer

# Normalise BOTH sides the same way before comparing.
norm = jiwer.Compose([
    jiwer.ToLowerCase(),
    jiwer.RemovePunctuation(),
    jiwer.RemoveMultipleSpaces(),
    jiwer.Strip(),
    jiwer.ReduceToListOfListOfWords(),
])

refs = ["Please book a table for two at seven.", "My account number is 4 4 1 9."]
hyps = ["please book the table for at seven thirty", "my account number is 4419"]

out = jiwer.process_words(refs, hyps,
                          reference_transform=norm, hypothesis_transform=norm)
print(f"WER={out.wer:.3f}  S={out.substitutions} D={out.deletions} "
      f"I={out.insertions} hits={out.hits}")
print(jiwer.visualize_alignment(out))   # REF / HYP lines with S, D, I markers

# Corpus WER pools all edits over all words. It is NOT the mean of per-utterance WERs.

Notice the second example. "4 4 1 9" vs "4419" is the same account number, but it counts as 4 substitutions-and-deletions unless your normalizer converts numbers to one canonical form. Real ASR evals use richer normalizers (for example, the English text normalizer shipped with OpenAI's Whisper repository, which handles numbers, spelling variants and fillers). Always report which normalizer you used; two WER numbers with different normalizers are not comparable.

Vendor guidance gives rough targets (Hamming, for example, describes under 5% WER as enterprise-grade and 5-10% as good), but treat those as vendor claims, not standards. What matters is WER on your audio: your callers' accents, your phone lines, your vocabulary. A model with low WER on clean read speech benchmarks can be much worse on noisy, accented 8 kHz calls.

Component level: the voice (TTS quality)

For the mouth, the classic measure is MOS, the mean opinion score. Human listeners rate each clip on a 1-5 scale (1 bad, 5 excellent), typically for "naturalness", and you average the ratings. The method comes from telephony (ITU-T Recommendation P.800 describes absolute category rating for speech quality). Related human tests include CMOS (comparative MOS, rating which of two clips is better) and MUSHRA-style tests that compare several systems side by side against a reference.

Human MOS studies are slow and expensive, so the field built automated MOS predictors: neural networks trained on thousands of human ratings to predict the score a listener would give. UTMOS, from the University of Tokyo's Saruwatari lab, is a widely used one: it was the UTokyo-SaruLab system for the VoiceMOS Challenge 2022 and placed first on 10 of the 16 challenge metrics. A reimplementation packaged for torch.hub (SpeechMOS) makes it a two-line check:

import torch, librosa

# Load a clip your TTS produced (mono). UTMOS predicts a naturalness MOS on a 1-5 scale.
wave, sr = librosa.load("agent_turn_017.wav", sr=None, mono=True)
predictor = torch.hub.load("tarepan/SpeechMOS:v1.2.0", "utmos22_strong", trust_repo=True)
score = predictor(torch.from_numpy(wave).unsqueeze(0), sr)
print(float(score))   # e.g. ~3.8; compare TTS voices / versions on the SAME sentences

What MOS misses: names, numbers and "reading" errors

For agents, the most damaging TTS failures are not about smoothness. They are text normalization failures: how the TTS turns written tokens into spoken words.

A neat, automatic way to test this is a round trip: synthesise the agent's sentence, transcribe it with a strong ASR model, and check that the entities survive (compute WER or exact entity match between the text you fed in and what came back). It does not measure beauty, but it does catch mispronounced and garbled content at scale. Keep a hand-curated "pronunciation set" of your hardest real names, product terms and number formats, and run it on every TTS or voice change.

Component level: latency

Latency in voice is not one number. It is a chain of delays, and the user only feels the total. The key user-facing metric is the voice-to-voice response latency: the time from when the caller stops speaking to when they hear the first sound of the reply. A closely related server-side measure is time to first audio (TTFA, sometimes "time to first byte", TTFB): the time from when the system decided the turn ended to when the first audio byte leaves the TTS.

Three lessons hide in that picture:

1. Endpointing dominates. A VAD that waits 500 ms of silence before declaring "turn over" adds 500 ms to every single reply. Shrink it and you cut latency, but you also start interrupting people who pause mid-sentence. That trade-off is why semantic endpointing exists: OpenAI's Realtime API offers a semantic_vad mode that "uses a semantic classifier to detect when the user has finished speaking, based on the words they have uttered", alongside a silence-based server_vad with a tunable silence_duration_ms. Open models such as Pipecat's Smart Turn do the same job for cascades.
2. Measure where the user is. Server logs start the clock at "VAD end", which already hides the endpoint wait. Put a recorder on the caller's side of the line, or record both channels of the call, and measure from the caller's last speech to the agent's first audible sound.
3. Report percentiles. Averages hide the awful calls. Report p50, p90, p95 and p99. Vendors publish rough ranges (Hamming reports industry end-to-end latency around 1.4-1.7 s at p50 and 3.3-3.8 s at p95; ElevenLabs suggests aiming for under 500 ms time to first audio and using p99 as the headline SLA), but treat these as vendor claims and set your own targets from user tests.

A latency harness sketch

The simplest reliable approach: stamp a monotonic clock at every stage boundary, per turn, and compute differences afterwards. Hook these into your pipeline's callbacks (every framework exposes events like "user stopped speaking" and "first audio frame out").

import time, statistics
from dataclasses import dataclass, field

@dataclass
class TurnTimer:
    """Timestamps for one conversational turn, in seconds (monotonic clock)."""
    turn_id: int
    marks: dict = field(default_factory=dict)

    def mark(self, event: str):
        # Only the FIRST occurrence counts (e.g. first TTS byte, not every chunk).
        self.marks.setdefault(event, time.perf_counter())

    def ms(self, start: str, end: str):
        if start in self.marks and end in self.marks:
            return (self.marks[end] - self.marks[start]) * 1000
        return None

# --- wire these into your pipeline callbacks -------------------------------
# on VAD "speech stopped"          -> timer.mark("vad_end")
# on final ASR transcript          -> timer.mark("asr_final")
# on first LLM token streamed      -> timer.mark("llm_first_token")
# on first TTS audio chunk         -> timer.mark("tts_first_byte")
# on first audio frame played out  -> timer.mark("first_audio_out")
# For a speech-to-speech API (OpenAI Realtime), the equivalents are the
# "input_audio_buffer.speech_stopped" event and the first "response.output_audio.delta".

def report(turns: list[TurnTimer]):
    def pct(vals, p):
        vals = sorted(v for v in vals if v is not None)
        if not vals:
            return float("nan")
        k = min(len(vals) - 1, round(p / 100 * (len(vals) - 1)))
        return vals[k]
    stages = {
        "ASR finalise":      ("vad_end", "asr_final"),
        "LLM first token":   ("asr_final", "llm_first_token"),
        "TTS first byte":    ("llm_first_token", "tts_first_byte"),
        "VAD end → 1st TTS": ("vad_end", "tts_first_byte"),
        "VAD end → audio":   ("vad_end", "first_audio_out"),
    }
    for name, (a, b) in stages.items():
        vals = [t.ms(a, b) for t in turns]
        print(f"{name:18s} p50={pct(vals,50):6.0f}  p90={pct(vals,90):6.0f}  "
              f"p95={pct(vals,95):6.0f}  p99={pct(vals,99):6.0f} ms")

# The user-perceived number needs the CALLER side: record the call in stereo
# (caller left, agent right) and measure from the caller's last voiced frame
# to the agent's first voiced frame with an offline VAD. That captures the
# endpoint wait and network delay that server-side marks cannot see.

Conversation level: turn-taking

Turn-taking is the part of voice evaluation with no text equivalent, and the part users complain about most ("it kept talking over me", "it cut me off", "it just went silent"). You can only see it on a timeline, so here is one call's timeline with the key events marked.

The turn-taking vocabulary

It helps to see interruption handling as a classification problem. Every time the caller makes a sound while the agent is speaking, the agent either yields or keeps going, and the caller either meant to take the turn or did not:

Task success, policy and tone

Above turn-taking sit the same questions as for any conversational agent (Module 06), asked of a spoken conversation:

• Task success. Did the agent achieve the caller's goal? Best graded on state, not words: was the appointment actually moved in the database, with the right date? τ-Voice (below) grades voice agents exactly this way, by checking the final database state.
• Policy adherence. Did it verify identity before discussing the account, read the required disclosure, refuse what it should refuse, never promise a refund it cannot give? These are mostly binary checks, graded by code on tool calls where possible and by an LLM judge on the transcript otherwise.
• Information capture. Did it collect and confirm critical fields (spell back the email, read back the phone number)? Voice makes confirmation loops essential because hearing is lossy.
• Tone and empathy. Did it acknowledge frustration, stay calm, avoid sounding robotic or sarcastic? This is where audio-aware grading matters, because the same words can be said kindly or coldly.

System level: call outcomes and business metrics

At the top of the stack are the numbers a contact-center manager watches. They are measured on real traffic (online evals, see Module 15) or on large batches of simulated calls.

Simulated callers: testing before real people call

You cannot ask a thousand real people to call your agent every time you change a prompt. So voice teams build simulated callers: an LLM plays a persona with a goal (the same user simulator idea from τ-bench in Module 06), a TTS voice gives it a voice, and it phones your agent over a real audio channel. This is voice-to-voice simulation: the agent under test hears synthesised speech, with all the ASR, endpointing and latency behavior that implies, instead of receiving clean text.

What a good simulated caller varies

• Goal and facts: what they want, and the account details they will give when asked (so the grader knows the right end state).
• Voice: accent, age, speaking rate, volume, pitch. Use several TTS voices; a suite with one synthetic voice tests one voice.
• Environment: background noise (street, café, car, TV, crying baby), speakerphone reverb, phone codec.
• Behavior: interrupts mid-sentence, pauses to think, corrects themselves ("the 14th, no sorry, the 15th"), gives information out of order, goes off topic, gets angry, stays silent.
• Adversarial intent: tries to get account details without verifying, social-engineers a refund, attempts a jailbreak.

Commercial platforms productise this loop. Hamming advertises auto-generated scenarios with "accents, interruptions, background noise, DTMF, and IVR trees" and turning production failures into regression tests; Coval describes simulating "thousands of realistic callers, hard policies, noisy audio"; Cekura describes persona simulation across language, accent, emotion, speed, volume, interruption behavior and background noise (their example multiplies 5 × 3 × 3 × 4 × 5, about 900 variants). These are vendor descriptions; evaluate any tool against your own calls. You can also build the loop yourself with an open framework (Pipecat or LiveKit Agents on both ends) and a TTS voice for the caller.

A scenario config

Whether you use a vendor or your own harness, write scenarios as data, so they can be versioned, reviewed, and expanded into a matrix. A sketch in YAML (the field names are illustrative, not a specific product's schema):

id: reschedule-dental-017
description: Caller wants to move a cleaning appointment; agent must verify identity first.

persona:
  name: "Marguerite Okafor"          # hard-to-pronounce name on purpose
  voice: { tts_voice: "caller_f_55_en_ng", rate: 0.9, volume_db: -6 }
  temperament: impatient
  behaviours:
    - interrupt_agent_when: "agent reads the policy disclaimer"
    - self_correct: "gives date as the 14th, then corrects to the 15th"
    - long_pause_ms: 4000            # looks for her card mid-call
  facts:                             # what she says ONLY when asked
    date_of_birth: "1970-03-22"
    current_appointment: "2026-10-09 09:30"
    wanted: "any weekday after 4 pm, week of 2026-10-12"

channel:
  noise: { type: cafe, snr_db: 10 }
  codec: g711_ulaw_8k
  packet_loss: 0.02

success_criteria:                    # graded by CODE on final state / tool calls
  - tool_called: verify_identity
    before: [lookup_appointment, reschedule_appointment]
  - db_state: "appointment.patient == 'Okafor' and appointment.start.weekday() < 5
               and appointment.start.hour >= 16 and appointment.start.date() == '2026-10-15'"
  - no_tool_called: cancel_appointment

judge_rubric:                        # graded by an LLM judge on the timestamped transcript
  - "Agent confirmed the new date and time out loud before ending the call."
  - "Agent stopped speaking within ~1 s when the caller interrupted the disclaimer."
  - "Agent used the correct date after the caller's self-correction."
  - "Agent re-prompted gently during the 4-second silence, without hanging up."

matrix:                              # expand into variants
  accent: [en-NG, en-IN, en-US-south, es-MX-L2]
  noise: [none, cafe_10db, car_5db]
  runs_per_variant: 3                # agents are stochastic; see pass^k in Module 04
max_duration_s: 240

Grading audio: transcribe-then-judge vs listening judges

Once you have recorded calls, how do you grade the qualities code cannot check? There are three approaches, and they differ in what they can "see".

What is lost in transcription

• Prosody: the melody of speech. "Great." said flatly and "Great!" said warmly are the same token.
• Emotion and paralinguistics: frustration, hesitation, sighs, laughter, sarcasm.
• Overlap and timing: a plain transcript serialises turns, so two people talking at once look like polite alternation, and a 4-second silence becomes invisible.
• Pronunciation: the transcript says "Siobhan" whether the agent said it right or wrong, because the ASR (or the agent's own text) already knows the spelling.
• Attribution of errors: if you judge the transcript of the caller side, ASR mistakes look like the caller said something odd, or worse, like the agent misunderstood when it actually heard correctly.

The practical fix is route B: produce a diarised (speaker-labeled), timestamped transcript from the stereo recording, and insert explicit event markers for overlaps, long silences, interruptions and tool calls. A text judge can then reason about timing. Leave prosody and empathy to an audio-capable judge or humans.

A judge prompt for a timestamped call transcript

You are grading a recorded phone call between a CALLER and a voice AGENT for a dental clinic.
You will receive a transcript with timestamps in seconds, speaker labels, and event markers:
  [OVERLAP a-b]   both speakers talking between a and b seconds
  [SILENCE n s]   no one speaking for n seconds
  [TOOL name(args) -> result]   a tool call the agent made
Transcripts come from automatic speech recognition and may contain recognition errors.
Do not penalise the agent for a CALLER-side transcription error unless the agent's
reply shows it actually misunderstood.

Grade each criterion independently as PASS or FAIL, with a one-sentence reason that
cites timestamps. Then give an overall verdict.

Criteria:
1. identity_verified_first: the agent verified identity (verify_identity tool succeeded)
   before revealing or changing any appointment details.
2. correct_final_booking: the final reschedule tool call matches what the caller last
   asked for (after any self-correction).
3. confirmation_read_back: before ending, the agent read back the new date and time and
   the caller did not dispute it.
4. yields_to_interruptions: whenever the CALLER starts speaking during an AGENT turn and
   it is a real attempt to take the turn (not "mm-hm"/"okay"), the overlap lasts under
   1.5 s and the agent's next turn addresses what the caller said.
5. no_premature_cut_in: the agent never starts speaking while the caller is mid-sentence
   (an OVERLAP that the AGENT started, where the caller's words are incomplete).
6. silence_handling: after any SILENCE over 3 s, the agent re-prompted politely within
   about 8 s and did not end the call unless silence persisted after two re-prompts.
7. no_fabrication: the agent never states availability, prices or policy that is not
   supported by a tool result or the policy text below.

Policy text:
<<POLICY>>

Transcript:
<<TRANSCRIPT>>

Respond with JSON only:
{"criteria": {"identity_verified_first": {"verdict": "PASS|FAIL", "reason": "..."}, ...},
 "overall": "PASS|FAIL", "first_failure_timestamp": null or seconds}

Notice the design choices, all from Module 13: one binary verdict per criterion (not a 1-10 vibe score), reasons that must cite evidence, an explicit warning about ASR noise, and a request for the first failure, which is what you need for error analysis. Anything that can be checked by code (criteria 1, 2, and the overlap duration in 4) should be checked by code as well; use the judge for the parts that need reading comprehension, and compare the two where they overlap.

Public voice benchmarks

Benchmarks tell you how capable an underlying model is at spoken interaction. They do not replace your own call suite, but they help you choose models and understand the state of the art. The main ones, as of this writing:

Also worth knowing by name: Artificial Analysis publishes speech-to-speech leaderboards (including Big Bench Audio results and latency measurements), and "Talking Turns" (Arora et al., ICLR 2025) trains a judge model specifically for turn-taking behavior. Other academic suites cover audio understanding and paralinguistics (for example SD-Eval for paralinguistic cues, and AIR-Bench and MMAU for general audio understanding); check their papers before relying on specifics.

Robustness: the channel is out to get you

A voice agent that works in a quiet office on a laptop microphone can fall apart on a real phone line. Robustness testing means deliberately degrading the audio in controlled, repeatable ways and watching which metrics move.

You can generate most of these conditions offline from clean recordings, which makes them repeatable (the same noisy clip every run):

import numpy as np
from scipy.signal import resample_poly

def add_noise(speech, noise, snr_db):
    """Mix noise into speech at a target signal-to-noise ratio (dB)."""
    noise = np.resize(noise, speech.shape)              # loop or trim noise to length
    p_speech = np.mean(speech ** 2)
    p_noise = np.mean(noise ** 2) + 1e-12
    scale = np.sqrt(p_speech / (p_noise * 10 ** (snr_db / 10)))
    return speech + scale * noise

def telephone_channel(x, sr=16000):
    """Narrowband phone line: 8 kHz sampling + G.711 mu-law companding."""
    x8 = np.clip(resample_poly(x, 1, sr // 8000), -1, 1)   # 16 kHz -> 8 kHz
    mu = 255.0
    enc = np.sign(x8) * np.log1p(mu * np.abs(x8)) / np.log1p(mu)
    q = np.round((enc + 1) / 2 * 255) / 255 * 2 - 1        # 8-bit quantisation
    dec = np.sign(q) * ((1 + mu) ** np.abs(q) - 1) / mu
    return resample_poly(dec, sr // 8000, 1)               # back to the model's rate

def packet_loss(x, sr=16000, frame_ms=20, loss_rate=0.05, seed=0):
    """Zero out random 20 ms frames, like lost RTP packets with no concealment."""
    rng = np.random.default_rng(seed)                       # fixed seed = repeatable
    n = int(sr * frame_ms / 1000)
    y = x.copy()
    for start in range(0, len(y), n):
        if rng.random() < loss_rate:
            y[start:start + n] = 0.0
    return y

# degraded = packet_loss(telephone_channel(add_noise(clean, cafe, snr_db=10)), loss_rate=0.02)

Offline augmentation tests the ASR and endpointing. It does not test your real telephony stack (SIP trunk, carrier codecs, jitter buffers, echo cancellation). For that, place real calls through the actual phone path in staging, at least for a smoke-test subset. Accents and speaking styles deserve their own slices: report WER and task success per accent group, because an average can hide a group the system fails badly (a fairness problem as well as a quality one). Consent matters here too: if you use recordings of real people, you need their permission to use their voice in tests.

Voice-specific safety

Everything from Module 17 applies to voice agents, plus a set of risks that only exist because the agent hears and speaks.

Putting it together: a voice eval plan

Here is how the pieces fit into one plan for a new voice agent, following the eval-driven loop from Module 12:

1. Collect real audio early. Even 50-100 recorded (consented) calls or role-played calls. Listen to them. Do error analysis on what goes wrong before choosing metrics.
2. Component suite (runs on every change, minutes): WER and entity accuracy on a fixed set of your callers' audio, sliced by accent and noise; a TTS pronunciation set with round-trip checks and automated MOS; per-stage latency percentiles.
3. Simulated-call suite (runs nightly or before release, hours): 50-500 scenarios × a persona matrix × several runs each. Grade with code (state, tool calls, timing), a timestamped-transcript judge, and a sampled audio or human review for tone.
4. Robustness and safety slices: the degradation ladder and a spoken red-team set, with thresholds that block release.
5. Production monitoring: containment, true resolution, escalation precision and recall, CSAT, latency percentiles and turn-taking rates on live calls, with judges sampling a share of calls. Every production failure you find becomes a new simulated scenario.

Check yourself

Sources and further reading

• Artificial Analysis, Big Bench Audio dataset and speech-to-speech leaderboard.
• Chen et al., VoiceBench: Benchmarking LLM-Based Voice Assistants (2024); leaderboard.
• Lin et al., Full-Duplex-Bench (2025).
• Scale AI, Audio MultiChallenge (2025); leaderboard.
• Ray, Dhandhania, Barres, Narasimhan, τ-Voice (2026); τ-bench leaderboard.
• Saeki et al., UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022; code; SpeechMOS (torch.hub).
• jiwer (WER, CER, alignment visualization) and its usage docs.
• OpenAI, Realtime API: voice activity detection (server_vad, semantic_vad) and server events.
• Google, Gemini Live API overview.
• Pipecat, Smart Turn (open semantic end-of-turn model).
• OpenAI, GPT-4o system card (voice-specific risks and mitigations).
• Hughes et al., Best-of-N Jailbreaking (2024), including audio language models.
• Stivers et al., Universals and cultural variation in turn-taking in conversation, PNAS (2009).
• Vendor guides (vendor claims, not standards): Hamming, voice agent evaluation metrics and interruption handling runbook; ElevenLabs, 6 pillars of voice agent evaluation; Cekura, voice AI evaluation metrics; Coval.