AI agent evaluation and QA, before launch and after it
Evaluation and QA for AI agents, ours or anyone’s: simulated users test the agent before launch, then run records, regular tests and cost control keep it right once it is live.
Most agents are tested with a few questions typed by the people who built them. Real users phrase requests differently and change their minds mid-conversation. An agent that worked at launch can also get worse. A document changes or the model is updated, and nobody re-runs the tests. Without measurement, nobody notices until a customer does, and costs grow unseen.
Simulated users test the agent, and every failure becomes a test
Before launch, simulated users hold multi-turn conversations with the agent through its endpoint. Each conversation is scored, and the report shows what broke. The agent itself does not change. Once it is live, we record every run, test it against your real cases and track its cost. We review the results with your team each month, and each failure becomes a new test case.
Systems it touches
- Your agent’s endpoint, whoever built it
- A record of every run
- Tests built from your real work
- Cost and usage reports for each agent
Example
A booking agent that gave out another customer’s slot
A made-up case that shows the mechanism. It describes no client.
- Your team has built a booking agent and wants it tested before launch.
- We describe the users: a regular customer, a caller who changes their mind, and someone asking about another person’s booking.
- Simulated users hold multi-turn conversations with the agent through its endpoint.
- The report shows one failure. When the request was rephrased, the agent gave out another customer’s booking time.
- Your team fixes the instruction. That conversation becomes a permanent test case, and the full run is repeated.
- Your test lead signs off the results before launch.
It works in the systems you already have
We test agents we built and agents others built, on provider models or open models. When a newer or cheaper model appears, the tests show whether it is good enough before anything is switched.
Where your data staysPermissions limit what it can touch
Where a person approves
- Your team signs off the test results before launch
- Changes to instructions, models or tools go live only after the tests pass
- Agreed limits send an alert to a person
- Run records stay in your own systems and follow your rules on how long data is kept.
- Every change has a score before and after, written into the review.
- If an agent falls below an agreed level, it goes back to asking for approval until it is fixed.
Everything we build is handed to your team
Who owns the code, and how handover works- A set of tests and simulated usersKept in your repository, so your team can run them again.
- A report of what brokeConversation by conversation, written for your team to keep.
- Dashboards and a monthly reviewRunning in your accounts. The review notes are yours.
How we would start
A test run first, then ongoing support
We start with a black-box test run against your agent and a report of what broke. Ongoing testing and monitoring is the running part of the Grow and run stage.
How we work togetherQuestions
Can you test an agent another team built?
Yes. The tests talk to the agent from the outside, through its endpoint, so nothing in it has to change. We report what we find before changing anything.
How do you keep model costs under control?
We measure the cost of each run and each result. Then we try cheaper options against the tests: smaller models, reused answers, shorter instructions. We keep a saving only when the quality holds.
What do you need from us to start?
Access to the agent’s endpoint or its logs, a short description of the people who use it, and a person on your side who owns the agent’s results.
Point us at the agent you want tested
Share its endpoint and describe the people who use it. We will plan a black-box test run and say what the report will cover.
Start a project