How to test voice AI: a 30-case plan for speech, actions and handovers

The short answer

Do not test a voice agent with improvised demo conversations. Use a fixed set of tasks, disruptions and expected outcomes, and score the conversation separately from the system action. A case passes only when the agent understands correctly, gives an allowed answer, performs the right action exactly once and hands over cleanly when uncertain.

A persuasive demo call shows one possible success path. A reliable test also shows how the system behaves under disruption, at its boundaries and when assumptions are wrong.

Why a demo call is almost always too kind

The person who built an agent knows the expected language. They speak more clearly, forgive pauses and unconsciously guide the call back to the intended route. A strong demo proves only that one designed path can work.

Production starts with uncomfortable cases: two intents in one sentence, a corrected phone number, sudden silence, a background speaker, a busy calendar, a repeated tool request, a question outside the knowledge base and a demand to speak with a person.

Current voice-platform documentation explicitly recommends testing unclear requests, configured actions, transfers, silence, interruptions and the closing sequence. OpenAI's Full Duplex Bench also covers pauses, turn-taking, background speech and backchannels. The resulting quality picture is broader than “the voice sounds good”.

The 30-case test matrix

The six areas receive five cases each. This is an editorial starting recommendation, not a scientifically optimal sample size. Five variants per area are usually enough to reveal recurring patterns while keeping acceptance testing practical.

Area A: core task

Test the direct success path, one missing required field, a correction, two plausible intents and an explicit cancellation. Write down the expected outcome and acceptable wording before the call.

Area B: speech and timing

Test a long thinking pause, a short acknowledgement such as “mm-hm”, an interruption, fast speech and background conversation. Do not score the beauty of the voice. Score whether the agent makes the right decision: keep listening, acknowledge briefly, stop or ask.

Area C: data

Use a difficult name, a spelled email address, a corrected number, an existing contact and two similar contacts. Inspect the record after the call. The transcript alone is not the source of truth.

Area D: actions

Test a successful action, a timeout, a retry, an already existing result and a missing prerequisite. A repeated request must not create a second appointment or task without checking.

Area E: boundaries and safety

Ask a question outside the knowledge base, demand an unapproved promise, object to data use, ask for a person and introduce a sensitive situation. The best result may be a polite refusal or handover.

Area F: routing and post-call work

Test business hours, out of hours, an unavailable destination, voicemail/fallback and a call that ends without booking. Verify the notification, task, summary and follow-up that should result.

Scoring with explicit denominators

Use four separate outcomes for each case. “Understood” means the intent and critical data were identified. “Answer correct” covers content and boundaries. “Action correct” requires exactly the expected system result. “Handover correct” includes the transfer or fallback and its context.

Do not collapse these levels into one attractive average. A call can sound excellent and still fail the action.

Stop criteria before launch

One critical failure can matter more than 29 successful cases. Do not launch with a wrong identity, prohibited statement, lost opt-out, confirmed but nonexistent booking, routing loop or missing human transfer in a defined escalation scenario.

Minor issues such as an unnecessary repetition are recorded but prioritised differently. This prevents an average score from hiding serious risk.

Regression testing after every change

Change one controllable component at a time: prompt, knowledge source, voice, interruption sensitivity or action. Rerun the affected cases and every critical safety case. Run the full matrix before a larger release.

Record the test date, agent version, model, voice, language, phone number, knowledge revision and calendar configuration. Without this context, two results cannot be compared fairly.

Measuring naturalness without false precision

Naturalness is subjective, but it can be reviewed consistently. Ask several testers four independent questions: Was I interrupted unnecessarily? Could I stop the agent? Were pauses and acknowledgements appropriate? Were critical names and numbers understandable?

OpenAI reports a 30-percentage-point Full Duplex Bench improvement for GPT-Live-1 over GPT-Realtime-2.1. This is a relevant vendor benchmark for interaction, but it does not replace testing in the actual German-language phone, data and process environment.

Production monitoring, not one-time approval

After launch, the test set becomes a control system. Review a regular sample for each call purpose. Turn new failure patterns into cases. Look beyond averages to critical failures and actions that people had to correct.

Accept a change only when it fixes the target issue without degrading established cases. A more natural conversation must not come at the price of less reliable data.

How ONE LOOP structures acceptance

ONE LOOP defines the responsibility, expected outcome, forbidden outcomes and handover for each agent. The 30 cases are checked against the voice layer, CRM, calendar and downstream workflow. An acceptance record shows which version was tested and which risks remain open.

Conclusion

Voice AI becomes dependable through a reproducible worst case, not the best demo call. A fixed 30-case plan is a practical balance: broad enough for speech, data, actions and fallbacks, yet small enough to run after every meaningful change. If an expected result cannot be described and verified in the system, it is not production-ready.