A voice agent eval is a repeatable test that measures an agent's behavior under realistic call conditions before it reaches live callers. Text-agent evals check what the model says; voice evals also have to check when it says it, whether it heard correctly, and whether the call ended in the outcome the business needed. Skipping them doesn't make an agent unevaluated — it makes your customers the eval.
The metrics that matter
- Task completion: did the call reach a verifiable end state — booked, triaged, escalated — not just a polite goodbye
- End-to-end latency at P95/P99: response gaps past roughly a second stop feeling like conversation; averages hide the calls that hurt
- Interruption handling: the agent should stop speaking within ~200ms of the caller barging in, then recover without losing context
- Transcription accuracy under real noise: word error rate on a quiet desk mic means little for a caller on a job site
- Hallucination rate: prices, availability, and policy answers the business never authorized
- Instruction adherence: the scoping rules — what the agent must never promise — hold under adversarial phrasing
Simulated conversations do the heavy lifting
You can't regression-test a phone agent by calling it yourself twice. The working pattern is a suite of simulated callers — scripted personas that interrupt, change their minds, mumble, and go off-script — run against the agent over live audio, with each conversation graded against a verifiable end state. Deterministic checks (was the booking created? was the handoff triggered?) anchor the suite; LLM-as-judge scoring covers tone and policy adherence where no deterministic check exists.
- Build the suite from real failed calls — every production failure becomes a permanent test case
- Run it on every prompt, model, or tool change; a voice agent without regression gates degrades silently
- Track metrics per scenario, not just in aggregate — an agent can improve on average while getting worse at emergencies
“Every deployment we run ends the same way: the eval suite is the deliverable. The agent is just the thing it measures.”
Where this fits
Evals are the final stage of the FDE roadmap because they're what separates a portfolio demo from deployment proof. To see the checks applied against real businesses, read the deployment walkthroughs — each one ends with the eval gate that change had to pass.