How to test an AI agent before it goes live
An agent's worst failures appear several turns into a conversation. We test agents by having simulated users talk to them, scoring each conversation, and letting the result decide the release.
By Sachi AIAI agents
Last updated:
An AI agent has to be tested by talking to it. Simulated users hold real multi-turn conversations with the agent. A judge scores each conversation, deterministic checks confirm what the agent actually did, and the result decides whether the change ships.
Unit tests miss the failures that matter in an agent
A unit test sends a fixed input and checks a fixed output. An agent breaks both halves of that contract. The same input does not always give the same output, and the conversation carries state from one reply to the next.
The failures that matter are conversational. A refund is approved without the customer being verified. Another customer's address is handed over. A refusal holds once and gives way when the user rephrases. None of these shows up in a single call. Each one appears only when somebody talks to the agent the way a real person does, over several turns.
Many agents also run inside low-code workflow tools or behind a webhook that the tester does not own, so there may be no codebase to instrument. The test should therefore treat the agent as a black box and need only its endpoint, with no change to the agent's code.
A green test suite can mislead in another way. A suite that runs against the source code never sees what was actually deployed, so it can pass while the running agent is broken.
A scripted persona drives each conversation
Each test is a persona with a goal, written as a few short fields:
- Persona: who the user is and how they behave, for example impatient and mildly rude.
- Context: what they know that the agent does not, such as an order reference.
- Goal: what the test designer is verifying, for example that a refund starts without the agent asking for the same detail twice.
- Opening message: the first thing the user says.
Keeping these fields separate lets you reuse them. The same goal, run by a patient user and by an impatient one, often surfaces different bugs.
A language model plays the persona. It is told to stay in character, answer the agent's questions from what it knows, and vary its wording the way a person would. A conversation ends in one of three ways: the simulated user signals that it got what it came for, it signals that the agent keeps missing the point, or the turn limit is reached. A simple similarity check helps too. If the user's new message is nearly the same as its last ones, the conversation is going round in circles and should be marked stuck.
Two habits keep runs honest. Every conversation gets a fresh session, so nothing leaks from one test into the next through the agent's memory. And for a fully repeatable run, a persona can carry scripted turns instead of generated ones.
The judge scores from the test designer's side
Once the conversations are saved, a second model judges them. The two phases are kept apart on purpose, so that a better scoring guide can be applied to the saved transcripts without talking to the agent again.
The judge's most important rule is whose side it is on. It scores from the test designer's point of view, never the simulated user's. If the agent correctly refuses to reveal another customer's data, that is a pass, even though the simulated user did not get what it asked for. A judge that scored it as a failure would push the team to weaken its own agent.
A scenario passes only when the judge says the goal was achieved, the score clears an agreed threshold and no blocking issue was found. The test designer decides in advance which severities block a release.
Deterministic checks confirm what the agent did
Beside the judge sit checks that do not depend on a model. They assert on what the agent did as well as what it said. If the agent's platform returns structured data with each reply, a check can read it: no basket was created unasked, the catalogue search tool was actually called, the account record was not changed.
Each check needs the right scope. A prohibition, such as never quoting a card number, applies to every turn. A requirement, such as confirming the delivery date, applies to the conversation as a whole. A common mistake is to check a requirement on every turn, which fails the scenario as soon as the user says "thanks". Severity should be declared by the test designer, never guessed by the tool.
A judge has to be checked before anyone trusts it
A model used as a judge is not perfectly consistent. The same transcript, scored twice with identical settings, can receive verdicts that disagree. Two defences help. Judge each transcript more than once and take the majority, adding votes when the first two disagree. And measure the judge against people.
To do that, a reviewer reads a sample of saved transcripts and labels each one pass or fail, without seeing the judge's verdict so as not to be led by it. Agreement is then measured with a statistic that corrects for chance, such as Cohen's kappa, because raw agreement misleads. Take a sample in which nine transcripts in ten deserve a pass. A judge that says "pass" to everything agrees with the reviewer nine times in ten and still tells you nothing about the failures. Its kappa is zero.
Until a judge has been checked against labels from your own agent's conversations, treat its scores as unvalidated.
Test data has to exist
If everything passes, check the test data before trusting the result. A persona asking about an order that does not exist in the agent's systems tests only the not-found path. Personas should be written from real requests, against records that are really there in the test environment.
Example. This is a hypothetical case to show the loop.
- A team adds a returns policy to its support agent and opens a change for review.
- The pipeline runs the persona suite against the test environment. An impatient customer asks for a refund on an order.
- The agent asks for the order number, gets it, then asks for it again. The simulated user marks the conversation stuck.
- The judge reports the goal as not achieved. The run fails.
- A person opens the transcript, sees the repeated question, and sends the change back before it ships.
The result feeds the release gate
A run should end in a result that a release pipeline can act on, and that result needs more than pass and fail. Four outcomes per scenario cover it. Pass and fail mean what they say. Flaky means the scenario passed on some repeats and failed on others. Error means the agent could not be reached, so the run makes no claim either way.
Errors should be kept out of the pass rate and, by default, should not block the release. An unreachable test server is not a defect in the agent, and teams tend to switch off a gate that fails during outages. The affected scenarios are run again once the agent can be reached, so an error never counts as a pass.
Repeats handle the agent's non-determinism. Run each scenario several times and set a floor on the pass rate. Compare each run with the last accepted one, scenario by scenario. A scenario that always passed and now passes only some of the time is a regression, even if it never fails outright.
Safety follows the same line. These conversations are real, so a refund the test starts is a real refund. Point the suite at a test environment, refuse production unless someone opts in deliberately, and show an estimate of the model cost before every run.
What this means for a buyer
A demo tests one conversation. A release decision needs personas written from real requests, deterministic checks on what the agent did, a judge checked against a person's labels, and a gate that fails on real defects and stays quiet during outages. Ask anyone building an agent for you, us included, to show those four things and a transcript of a failure they caught.
This is the method we use to test an agent before launch and to keep checking it once it is live. The agent evaluation and QA page describes that work.