How to Evaluate AI Output: Building Evals That Catch Real Problems

Here is a situation that plays out on almost every team building with language models. Someone changes a prompt. The output looks better on the two examples they tried. It ships. A week later something unrelated is broken and nobody can say when it started, because there was never a baseline to compare against.

Evaluations — evals — are the fix. They are the equivalent of a test suite for a system whose output is not deterministic, and building even a crude one puts you ahead of most teams shipping AI features today.

Why "it looks good" fails

Eyeballing output has three problems, and they compound.

You test the happy path. People try the examples they had in mind when writing the prompt. Those are the cases it already handles. The failures live in input nobody thought about — the empty field, the request in another language, the customer being sarcastic.

You cannot compare. "Better" against what? Without a fixed set of inputs and a record of previous output, every judgement is made against a fading memory of how it used to behave.

Output is not deterministic. The same prompt produces different output each time. Run it twice, get two impressions, and you have measured sampling noise rather than a change.

Start with fifteen examples

The instinct is to build something comprehensive, which is why most eval projects never ship. Start smaller than feels serious.

Collect fifteen to twenty real inputs. Not invented ones — real ones, from logs, from support tickets, from whatever your users actually send. Include:

• Five typical cases, the bread and butter
• Five awkward ones: ambiguous, badly worded, missing information
• Three that have already gone wrong in production
• Two that should be refused — out of scope, or something the system should decline

That last group matters more than its size suggests. A system that answers everything confidently, including things it should decline, is a system that will eventually answer something it should not.

For each input, write down what a good response looks like. Not the exact words — the properties. "Names the refund window", "does not invent a policy", "asks for the order number if it is missing".

Three ways to grade

Deterministic checks are the cheapest and the most underused. Is the output valid JSON? Does it contain the required fields? Is it under the length limit? Did it avoid the forbidden phrases? These catch a surprising share of real failures and cost nothing to run. Start here.

Model-graded evaluation uses a second model to judge the first. It scales to things a rule cannot check — is this accurate given the source, does it answer the question asked, is the tone right. It works better than most people expect, with two conditions: give the grader a specific rubric rather than asking "is this good", and check the grader against human judgement on a sample before trusting it. A grader that disagrees with you on the cases you care about is worse than no grader.

Human review stays necessary for anything subjective and for spot-checking the other two. It does not need to be constant — reviewing twenty outputs a week catches drift long before users report it.

What to actually measure

Vague metrics produce vague results. Useful ones are specific to the failure you are worried about:

Task completion. Did it do the thing? Binary, per case. This is the one that matters most and the one people skip in favour of quality scores.

Grounding. For anything retrieval-based: is every claim supported by the retrieved material? Unsupported claims are the failure mode that damages trust fastest, because they are fluent and confident.

Refusal accuracy. Does it decline what it should decline, and — the other half people forget — does it answer what it should answer? A system tuned to refuse aggressively looks safe on one metric and useless on the other.

Format compliance. Did it return the shape the downstream code expects? Deterministic, cheap, and the most common cause of production errors.

Regression. Did anything that worked last week stop working? This is the whole point of having a fixed test set.

Run them when it matters

An eval suite nobody runs is documentation. Wire it into the moments where a change happens: when a prompt is edited, when a model version changes, when the retrieval pipeline is touched, and on a schedule to catch provider-side drift.

That last one is not paranoia. Providers update models behind the same version string, and behaviour shifts without an announcement. A weekly run against a fixed set is how you find out before your users do.

The uncomfortable part

Your first eval run will produce a worse score than you expect. This is not a sign the evals are wrong. It is the first honest measurement of something that was previously assessed by vibes.

That is uncomfortable and it is the entire value. A number you can move beats a feeling you can argue about, and it is the only way to know whether the change you just made helped.

If you are building AI agents that run many steps, evals stop being optional. Per-step reliability compounds — a system that is 95% reliable per step is under 60% reliable across ten — and there is no way to see that without measuring it.