AI agent evaluation is the part everyone skips. You wire up a model, a few tools, a system prompt, and the demo works. It answers the three questions you tried. You ship it. Two weeks later a customer asks something slightly off-script and the agent confidently invents a refund policy that doesn't exist.
I've built and shipped AI features that real users pay for, and the thing that separates a demo from a product is not the model, the framework, or the prompt. It's whether you can answer one question with evidence: how often does this thing get it right, and how do you know?
Most teams can't answer that. LangChain's State of Agent Engineering survey found that while ~89% of teams have some observability on their agents, only about half run evals at all. That gap is the whole problem. Observability tells you the agent responded in 900ms. It doesn't tell you the response was wrong.
Why your test suite doesn't work here
Traditional tests assert equality. expect(total).toBe(42). That model breaks the moment output is generated text, because there are a thousand correct phrasings and one subtly catastrophic one that differs by a single word.
Worse, agents are multi-step. A failure at step four might trace back to a bad tool-selection decision at step one. Your integration test says "the agent responded" — green check, shipped, broken. And the environment moves underneath you: APIs go down, your database changes between queries, and the correct answer depends on what's true right now.
Non-determinism is the other half. Run the same input twice, get two different paths. A single passing run proves nothing. You need distributions, not assertions.
Start with failure collection, not metrics
The instinct is to reach for a scoring framework first. Don't. The highest-leverage thing you can do in week one is build a habit of reading your agent's actual output.
Log every interaction — full input, every tool call and its arguments, intermediate reasoning, final output. Then sit down and read fifty of them. Not skim. Read.
This is unglamorous and it is the entire job. Teams that build reliable agents spend the bulk of their development time understanding failures rather than writing automated checks. You cannot write a good eval for a failure mode you haven't seen yet, and the failure modes are never the ones you predicted. When I was building the AI interview coach inside Intavue, my assumption was that the hard failure would be technical accuracy — the model getting a data-structures answer wrong. It wasn't. The real failure was tonal: the coach would diagnose a weak concept correctly and then explain it in a way that made the user feel stupid. No accuracy metric on earth catches that. Reading transcripts does.
Group what you find into named failure modes. "Hallucinated a policy." "Called the search tool when it should have asked a clarifying question." "Correct but three paragraphs too long." Those names become your eval suite.
Build the smallest dataset that hurts
Now take those failure modes and turn them into a fixed set of test cases. Thirty to fifty is plenty to start. Every one should be a real input that either broke the agent or came close.
Keep it versioned in your repo next to the code, as plain JSON or YAML. Every case gets the input, any state the agent needs, and what a good response looks like — which is usually not an exact string but a set of criteria: must not invent a policy, must cite a source, must ask a clarifying question if the budget is unspecified.
The value here compounds. Every production bug becomes a permanent test case. Six months in you have a suite that encodes everything you've learned about how your specific agent fails, which is worth far more than any public benchmark. Public benchmarks are actively misleading now — UC Berkeley researchers showed in 2026 that major agent benchmarks can be gamed to near-perfect scores without solving a single task, and there's a growing body of work on just how unreliable agent scoring is. Your thirty cases beat a leaderboard score every time.
Grade with code first, models second
For each criterion, ask whether code can check it. A surprising amount can:
- Did it call the right tool with the right arguments? Assert on the trace.
- Is the JSON valid against the schema? Parse it.
- Did it stay under the token budget, latency target, cost ceiling? Measure it.
- Did it leak a customer ID into the response? Regex it.
Deterministic checks are free, fast, and never flake. Exhaust them before reaching for anything cleverer.
What's left — tone, helpfulness, faithfulness to retrieved context — is where you use a model as the judge. Give the judge the input, the output, and one specific yes/no question. Not "rate this 1-10," which produces noise dressed as data. "Does this response contain any claim not supported by the provided context? Answer yes or no with the offending sentence." Narrow questions get consistent answers.
Then validate the judge. Hand-label thirty cases yourself, run the judge against them, and measure agreement. If your judge disagrees with you 30% of the time, you've built a random number generator with a confidence problem.
Cost is the reason to order it this way. A human review runs $5–50 per instance; an LLM judge costs fractions of a cent. But an unvalidated judge costs you a false sense of safety, which is more expensive than either.
Run AI agent evaluation in CI, then in production
Wire the suite into CI so prompt changes are gated the same way code changes are. A prompt edit is a deploy — treat it like one. Track the pass rate per failure mode over time, because the aggregate number hides regressions: overall 94% looks stable while your hallucination category quietly slid from 98% to 71%.
Then take the same graders and run them on a sample of live traffic. Offline evals catch regressions you anticipated; online evals catch the distribution shift you didn't. Real users ask things your dataset never imagined, and that's how new cases get discovered. It's the same loop I described in Why Your RAG Pipeline Is Probably Broken — retrieval quality degrades silently, and only measurement on live data surfaces it.
Sample it. Grading 100% of traffic with a frontier model gets expensive fast. Five percent, weighted toward high-value or high-risk flows, tells you what you need.
What this buys you
The payoff isn't a dashboard. It's that you can change things. Swap models, rewrite a prompt, add a tool, cut costs by routing simple queries to a smaller model — and know within minutes whether you broke anything. Without evals, every change is a gamble and the rational move is to freeze the system, which is how AI features rot.
It also changes how you sell. Voice agents live or die on this — I wrote about the production realities of shipping voice AI separately, and the same principle applies: buyers in 2026 have seen enough glossy demos. Saying "our agent resolves 84% of tier-one tickets, measured against 200 labeled cases, here's the methodology" closes deals that a demo doesn't.
If you're standing up an AI feature and this loop doesn't exist yet, that's the thing to build first — before more features. You can see the rest of what I've shipped in production if you want the longer version, or reach out and I'll tell you where your agent is most likely breaking.