Agent Shakedown

How to Test a Voice Agent Before Launch

Testing a voice agent is not the same as testing a chatbot. A chatbot waits politely for the next message. A voice agent lives in real time, against a person who interrupts, mumbles, corrects themselves, and says yes before the sentence is finished. Most pre-launch testing misses that, which is why most agent problems are discovered by customers.

This is the checklist we use to test voice agents before launch. It works on any platform — Retell, Vapi, or a custom stack — because every step is done from the caller's side.

Step 1: Write down the job before you test anything

You cannot score an agent against a job that only exists in your head. Before the first test call, write, in plain language:

  • What the agent must accomplish on a successful call
  • What it must always say (prices, fees, disclosures, policies)
  • What it must never do (grant exceptions, quote prices it can't know, promise timelines it can't keep)
  • What it should do when it can't complete the job

This becomes your scoring sheet. If a behavior isn't written down, you will argue about it later instead of measuring it.

Step 2: Run the happy path — then stop trusting it

Make the call that goes perfectly: the caller wants the normal thing, gives information in order, and agrees to everything. Your agent should complete the job end to end, deliver every required statement, and close cleanly.

Run it three times, not once. Voice agents are not deterministic, and a flow that works once may wander on the third run. Then move on — the happy path is the least informative test you will run, and it is where most self-testing stops.

Step 3: Test the information, not just the flow

Real callers hand over information badly. Run calls where the caller:

  • Gives details out of order — address before name, problem before either
  • Corrects themselves mid-call ("Tuesday — no, Thursday")
  • Gives partial information and has to be asked twice
  • Spells nothing, assumes the agent heard correctly, and is wrong

Score what the agent writes down, not what it says. Compare the final record — name, address, phone, booked slot — against what the caller actually said. Transcription and capture errors hide here, especially in names, addresses, and numbers. When in doubt, the agent should read details back and confirm them; test whether it does.

Step 4: Interrupt it

This is the test most builders skip and most callers perform on day one.

  • Talk over the agent while it is delivering a required statement, and agree early ("yeah, that's fine")
  • Answer a question before the agent finishes asking it
  • Change your mind mid-sentence
  • Go silent for several seconds

A well-behaved agent stops talking when you talk, listens, adapts, and continues from where it makes sense. A badly behaved one restarts statements from the beginning, repeats itself, or steamrolls the caller. Interruption handling is a configurable, testable behavior — we wrote up a case study of an agent that restarted the same fee statement three times in one call: Why your agent repeats itself.

Step 5: Test the boundaries

Every agent has lines it may not cross. Test whether it holds them under pressure:

  • Ask for a discount, exception, or earlier slot it is not allowed to give
  • Ask it to do something outside its job entirely
  • Give it a situation it should escalate or close out — a safety issue, an angry caller, a request it has no answer for

The pass condition is not charm. It is that the agent declines or hands off cleanly, without inventing a policy, a price, or a promise.

Step 6: Measure latency as a percentile, not a feeling

"It felt fast" is not a test result. On each test call, record the time between the caller finishing a sentence and the agent starting to answer. Report it as a median (p50) and a 90th percentile (p90) across all your test calls.

Averages hide the problem: an agent that usually answers quickly but regularly makes callers wait through long pauses will feel broken, and the average will look fine. The p90 is where the bad experience lives. Re-measure after any change to model, transcription mode, or prompt length — all three move latency, and not always in the direction you expect.

Step 7: Score it, write it down, keep the recordings

For every test call, record a simple pass/fail and one line of evidence for each category on the scorecard: task completion, fee and policy delivery, interruption handling, latency, fallback behavior. Keep the audio and transcript for every scored call. A score with no recording behind it is an opinion, and opinions cannot be compared across runs.

The full scorecard, with pass conditions for each category, is here: AI agent evaluation checklist.

Step 8: Re-run the same calls after every change

A prompt edit, a model swap, a new voice — any of them can fix one behavior and break another. The value of a scripted test set is repeatability: the same scenarios, the same scoring, before and after. If you change the agent and don't re-run the set, you haven't tested the change. You've just made it.

Keep the test set small enough to run every time — ten to fifteen scenarios you actually run beats fifty you don't.

A minimal pre-launch pass, if time is short

If you do nothing else before launch, do these five calls:

  1. One clean happy-path call, scored against the job definition
  2. One call where you interrupt the required statement and agree early
  3. One call with out-of-order and corrected information, checking the final record
  4. One boundary call it must decline or hand off
  5. One call measuring response latency across every turn

Five calls will not certify an agent. They will catch the failures your customers would otherwise find first.

Agent Shakedown runs this process as a service — scenario runs and a scored report for voice agents on Retell, Vapi, or a custom stack. Launching soon: join the waitlist.