Every AI feature I have shipped hit the same wall around week three. The demo works, the founder is happy, and then someone tweaks a prompt to fix one bad output — and nobody in the room can say whether the other ninety-nine outputs got better or worse. Nobody had worked out how to test AI features whose answer changes on every run, so the project stalls. Not because the model is bad. Because the team cannot answer a basic question: did that change help?
This is the part that separates an AI demo from an AI product, and it is almost never the part anyone budgets for.
Your existing test suite quietly gives up here
expect(output).toBe("...") is dead on arrival. Temperature sampling means valid answers vary run to run, and you get drift even at temperature zero. So exact-match assertions fail on perfectly good responses and — worse — pass when a wrong answer happens to match the string you froze last month.
The reflex fix is to stop testing the AI parts and lean on "we'll eyeball it." That works for about two weeks. Then you have four prompts, two models, a retrieval step and a tool-calling loop, and eyeballing means one person spot-checking five outputs and declaring victory. I have watched teams ship regressions this way that took three weeks to notice, because the failure was a 15% drop in a quality no dashboard was measuring.
What replaces exact matching is not one clever trick. It is a set of graded checks run over a fixed dataset, scored statistically instead of pass/fail on a single run.
Build the golden set before you touch a framework
The first thing to build is not an eval library. It is a file of real examples.
When I built the code-grading engine inside Intavue — it runs and grades candidate solutions in JavaScript, Python, Java and C++ on approach and edge cases, not just whether tests pass — the eval work started as a spreadsheet of about sixty real submissions with a human verdict attached to each. Correct-but-slow. Correct-but-unreadable. Passes the tests and completely misses the point. Those edge cases were the whole product, and none of them came from my imagination. They came from watching real output.
Practical rules for the golden set:
- Pull from real traffic, not synthetic prompts. Log every input and output from day one. Your first fifty production failures are worth more than five hundred generated examples.
- Label the failure, not just the score. "Bad" is useless. "Ignored the resume context" and "hallucinated a library that does not exist" are two different bugs with two different fixes.
- Keep it small and keep it hard. Forty examples you actually curated beat four hundred you skimmed. Weight it toward the cases that break things.
- Freeze it. The dataset, the prompts, the model version, the evaluator version. If you change the ruler while measuring, the measurement is worthless.
Sixty examples is enough to catch real regressions. I have never needed thousands.
How to test AI features in three layers of checks
Build them in this order, because each layer is cheaper and more reliable than the next one.
1. Deterministic assertions. Boring and undefeated. Is it valid JSON? Does it match the schema? Are all cited source IDs actually in the retrieved context? Is it under the token budget? Did it call the right tool with the right argument types? A surprising share of "the AI is unreliable" complaints are structural failures that a schema check catches in a millisecond for zero cost.
2. Property checks. Not "is this the right answer" but "does this answer have the properties it must have." Groundedness — is every factual claim traceable to retrieved context? Refusal behavior on out-of-scope input. No leaked system prompt. Correct language. These catch the failure modes I wrote about in why your RAG pipeline is probably broken, where retrieval quietly degrades and the model fills the gap by inventing things.
3. LLM-as-judge. Only for the genuinely subjective stuff — tone, helpfulness, whether an explanation actually teaches the concept. It is the expensive, noisiest layer, so it should be the thinnest one. If you are judging things layer one could have asserted, you are burning money and adding variance for nothing.
Most teams start at layer three because it feels like the AI-native answer. Then their eval suite is slow, costs real money per run, and has its own error bars. Start at the bottom.
Your judge is a feature, and it needs its own tests
The mistake I see constantly: someone writes "Rate this response 1-10 for quality" and treats the number as ground truth. That prompt will hand out 7s and 8s to almost anything, and the score will drift the moment you change models.
What works better:
- Binary or three-point verdicts, not 1-10. "Does this answer ground every claim in the provided context? yes/no." Models are far more consistent on narrow judgments than on numeric scales.
- A rubric with examples. Show the judge two passing and two failing responses with reasoning. Same thing you would give a human grader.
- One dimension per judge call. A single call scoring accuracy, tone and completeness collapses into a vibe score. Split them.
- Validate the judge against humans. Take thirty examples you labeled by hand, run the judge, measure agreement. If it agrees with you 60% of the time, it is not measuring your product — it is adding noise. Fix the rubric until agreement is high, then trust it.
That last step is the one everybody skips, and it is the one that makes the whole system worth running. Hamel Husain's post on evals is the best free writeup on this loop, and OpenAI's evaluation guide covers the mechanics well.
Run it in CI, on the diff, every time
An eval suite nobody runs is a folder of dead code. Wire it to the same trigger as your unit tests: any PR touching a prompt, a model version, a retrieval parameter or a tool definition runs the suite and posts scores in the PR.
Two things make this bearable. Cache aggressively — same input, same model version, same prompt hash means you replay the stored result instead of paying for it again. And set thresholds as deltas, not absolutes: fail the build on a drop of more than a few points from main, not on some arbitrary 90% bar. Absolute thresholds get muted within a month.
For the real-time voice interviewer in Intavue, the evals run against recorded audio sessions rather than live calls — deterministic input, measurable output, no flakiness from network conditions. The rest of what that pipeline took is in what it actually takes to ship a voice AI agent in production.
Production is where the real eval set comes from
Offline evals stop regressions you already know about. Production tells you what you never thought to test.
Sample a slice of live traffic through your cheap deterministic and property checks continuously — schema validity, groundedness, refusal rate, latency, cost per request. Alert on the rate, not on individual failures. And make it one click for a user or a support agent to flag a bad response, because every flagged response is a golden set entry someone else paid you to find.
Then close the loop: flagged output becomes a labeled example, the example goes in the golden set, the golden set catches the regression next time. That is the entire game.
What this actually costs
A working eval harness for a single AI feature is a few days of engineering, not a quarter. The golden set is the slow part, and it is slow because it is judgment work, not code.
What you get is the ability to change a prompt on a Friday without holding your breath, swap models when a cheaper one lands, and answer "is it getting better?" with a number instead of a feeling. Teams that ship reliable AI products are not the ones with the cleverest prompts. They are the ones who can measure a change in ten minutes instead of arguing about it for a week.
If your AI feature works in the demo and nobody can prove it works in production, that is the gap — and it is a fixable one.