AI Agent Evaluation Checklist: The Pre-Production Scorecard

Most agent evaluation fails for a boring reason: there is no scorecard. The builder makes a few calls, the agent sounds reasonable, and it ships. Then production supplies the test cases — with real customers in them.

This checklist is the scorecard we use to evaluate agents before production. Five categories, each scored the same way on every run, each backed by a recording or transcript. It applies to chat and workflow agents too, but it was built voice-first, because voice is where the failures are hardest to hide.

Print it, copy it, run your agent against it.

1. Task completion

What it measures: Did the agent finish the job it exists to do?

How to score it: Define the job in one sentence before testing — "book a service appointment with a confirmed name, address, phone, and time window." Run scenarios covering the normal path and the common variations. For each run, score: completed in full / completed with errors / not completed.

Pass conditions:

• The end state is complete: every required field captured, every required step done
• Captured data matches what the caller actually said — check the record, not the agent's summary
• The agent confirms the outcome back to the caller before the call ends

Common failures: the agent books without confirming; the agent silently drops a field it failed to capture; the agent reports success on a task it only partially finished.

2. Fee and policy delivery

What it measures: Did the agent state the required prices, terms, and disclosures completely — and exactly once?

How to score it: List every statement the agent is required to deliver: service fees, what the fee covers, whether it's credited toward work, cancellation terms, recording notices, anything a customer must hear before agreeing. In each run, score each statement: delivered in full / delivered partially / not delivered / delivered more than once.

Pass conditions:

• Every required statement delivered in full before the caller is asked to agree
• The caller's agreement comes after the statement, not instead of it
• No required statement is repeated — repetition is a defect, not thoroughness (see the case study: Why your agent repeats itself)

Common failures: the fee is mentioned but what it covers is not; the agent skips the disclosure when the caller seems in a hurry; the agent restarts the statement when interrupted and delivers it twice.

3. Interruption handling

What it measures: What happens when the caller talks over the agent — because they will.

How to score it: In dedicated test runs, interrupt the agent at planned points: during the greeting, during a required statement, mid-question. Also answer questions early, change an answer mid-sentence, and go silent once. Score each event: agent stops and adapts / agent continues after a delay / agent restarts or repeats / agent talks over the caller.

Pass conditions:

• The agent stops speaking promptly when the caller speaks
• It never restarts a statement from the beginning after an interruption
• If the caller agrees or answers early, the agent treats it as the answer and moves on
• After an interruption, the conversation continues from a sensible point — the caller should never have to repeat themselves because the agent lost the thread

Common failures: restarting scripted lines from the top; treating an early "that's fine" as nothing and re-delivering the whole statement; answering a question the caller didn't finish asking, wrongly.

4. Latency (p50 / p90)

What it measures: How long the caller waits, in silence, for the agent to respond.

How to score it: For every turn in every test call, measure the gap between the caller finishing and the agent starting. Report two numbers across all turns: the median (p50) and the 90th percentile (p90). Do not report a single average.

Pass conditions:

• You have the numbers — "felt responsive" is not a measurement
• The p90 is within the range your callers will tolerate without "hello, are you there?"
• Latency is re-measured after any change to model, transcription settings, voice, or prompt length, and compared against the previous run

Common failures: a good average hiding a bad tail; a change that improved wording and quietly doubled response time; long pauses that callers fill by repeating themselves — which then triggers interruption failures on top of latency ones.

5. Fallback behavior

What it measures: What the agent does when the conversation leaves the script — the request it can't fulfill, the input it can't parse, the situation it should not handle.

How to score it: Run scenarios the agent must not complete: an out-of-scope request, a caller it cannot understand after repeated attempts, a situation that requires a human, a request it is forbidden from granting. Score: clean handoff or close / degraded but honest / failed silently or fabricated.

Pass conditions:

• The agent says it can't do that, plainly, instead of improvising
• It offers the defined fallback: a callback, a handoff, a message taken — whatever the job definition specifies
• It never invents a price, a policy, a capability, or a confirmation to escape an awkward moment

Common failures: hallucinating an answer to end the call; looping the same question; agreeing to something outside its authority.

How to use the scorecard

1. Write the job definition first. Every category scores against it. No definition, no score.
2. Run the same scenario set every time. A small set run consistently — across prompt changes, model changes, settings changes — is what turns opinions into comparisons.
3. Keep the evidence. One recording or transcript per scored run, linked to the scores. A score you can't re-examine is a score you can't defend.
4. Set your pass bar before you run. Decide in advance what ships and what doesn't — for example, no launch with a fallback fabrication, or a repeated required statement, at any score. Bars set after the results come in tend to move.

The scorecard, in one view

If you are preparing a voice agent specifically, the step-by-step version of this process is here: How to test a voice agent before launch.

Testing a personal agent

Personal agents — the assistants people run for themselves, like Grok, Meta AI, Dots, or Duet — don't take customer calls, but the same scorecard applies once you translate the categories:

• Task completion becomes errand completion: did it actually send the message, book the thing, set the reminder — end to end — or did it just say it would?
• Fee & policy delivery becomes confirm-before-acting: did it check before spending money, sending something, or changing something it can't undo? Score both directions — the agent that acts without asking, and the agent that asks about everything.
• Required statements becomes discretion: did anything it knows leak into the wrong conversation, the wrong person, or the wrong moment? For a personal agent this is the category that must score perfectly.
• Interruption handling becomes correction handling: when the owner changes their mind mid-task, does it adapt, or does it finish the wrong errand anyway?
• Fallback behavior stays the same: admit the limit plainly instead of fabricating.

Personal agents also need one test class business agents don't: memory probes. Tell the agent a fact in one conversation; ask for it days later, phrased differently. Repeat across weeks. An assistant that forgets — or confidently remembers something that was never said — fails the job it was hired for.

Agent Shakedown runs this scorecard as a service — scenario runs against your agent on Retell, Vapi, or a custom stack, delivered as a scored report with recordings behind every finding. Launching soon: join the waitlist.