Testing AI before it reaches your customers

AI features fail differently from normal software. Here is what an evaluation suite checks, and how it fits into a normal release cycle.

Why normal testing isn’t enough

A normal feature either works or it doesn’t: you click the button and the order is placed, or it isn’t. An AI feature can give a slightly different answer each time, sound confident when it is wrong, and get worse after an update you didn’t make, because the model provider changed something.

So you can’t test it once and move on. You need a repeatable way to measure how good it is, and to notice when that changes.

Build an evaluation set from real cases

The core of AI testing is an evaluation set: a few hundred real examples, each with the answer you expect. For a support assistant, that might be past customer questions and the replies your best agents sent. For document extraction, it is a pile of real invoices with the correct fields filled in.

Include the awkward cases: misspellings, two questions in one message, documents scanned upside down. That is where AI features fail in production.

What we measure

For each release we track accuracy against the evaluation set, how often the system says it doesn’t know (often the right answer), response time and cost per request. We also run adversarial prompts: attempts to make the assistant ignore its instructions, reveal data it shouldn’t, or promise something the business can’t deliver.

Every change to a prompt, a model or a data source runs the full suite before it ships, the same way unit tests run on normal code.

Keep watching after launch

Testing doesn’t stop at release. We log a sample of real outputs, review them every week and add new failure cases to the evaluation set.

After a few months that set becomes one of the most valuable parts of the project. It tells you whether a new model is actually better before you switch to it.