An agent that handles ten hand-picked examples perfectly tells you almost nothing. We've watched agents ace a demo on Tuesday and then, on Thursday, confidently refund the wrong customer because a ticket mentioned two order numbers. Nobody had tested that case. Nobody had thought of it.
AI agent evaluation is how you find Thursday's failure on Tuesday. It means running the agent against a fixed set of real cases, scoring what it did against what a good operator would have done, and repeating that every time anything changes. Prompt, model, tool, or data. Any of them can break behavior that worked yesterday.
Most teams skip this step. They test by vibes: someone tries a few inputs, the answers look sensible, and the agent goes live. When it breaks, there's no record of what "working" meant.
Below: how to build a test set from your own operation, what to score, how to grade it, and how to keep testing after launch. You don't need an ML team. You need a spreadsheet, a few hours from the people who do the work today, and the discipline to rerun it.
What is AI agent evaluation?
AI agent evaluation is the practice of testing an AI agent against a fixed set of realistic cases with known good outcomes, then scoring its results, its actions, and its judgment about when to stop and ask a person. It turns "seems fine" into a pass rate you can track across every change.
Two words in that definition carry the weight. Fixed means the same cases run every time, so a score of 90% this week and 80% next week tells you something changed. Actions means you check what the agent did along the way, and not only the final message. An agent can write a perfect reply to a customer after updating the wrong record.
An AI agent can take a different path on the same input from one run to the next. That's the flexibility you pay for, and it's why one-input, one-output integration testing doesn't cover it.
Expert tip: write down what a good operator would do for each case before you run the agent. If you look at the agent's answer first, you'll grade generously.
Why demos lie about agents
A demo is a test set of five cases chosen by the person who wants the project approved. Of course it passes.
Real work has a long tail. The customer who writes in two languages. The invoice where the PO number is in the email body instead of the attachment. The request that's half refund, half complaint. In our experience, these odd cases are a small share of volume but a large share of the damage, because they're exactly where an agent guesses.
And agents fail quietly. Rules-based automation usually breaks loudly: a field is missing, the job errors, someone gets an alert. An agent that misreads a case still produces a fluent, confident answer. Nothing in the output looks wrong. We covered this gap in AI agents vs automation, and it's the main reason evaluation matters more for agents than for anything else you'll build.
Anthropic's guide to building effective agents makes the same point from the builder's side: autonomy means errors can compound, so agents need extensive testing in a sandbox before they touch real systems.
There's a second trap. Agents change under you. A model provider ships an update, someone tweaks a prompt to fix one complaint, a tool's API returns a new field. Each change is small. Without a test set, you find out what it broke from a customer.
Build an agent test set from real work
The test set is the whole game. A bad one gives you a high score that means nothing.
Where the cases come from
Pull them from your own history, not from imagination. Good sources, in rough order of value:
- Recent real cases. Take a random slice of the last few months of the work the agent will do. Random matters. Hand-picked cases skew easy.
- The cases your team remembers. Ask the operators which ones were painful. They'll name them in minutes, and those are the cases that will hurt in production.
- Past escalations and errors. Anything that went to a supervisor, got reversed, or caused a complaint.
- Cases the agent should refuse. Requests outside its scope, missing information, anything that needs a human decision by policy.
That fourth group gets skipped constantly. If your agent is designed with an operator in the loop, the test set has to check that it actually hands off when it should.
How many cases you need
Fewer than you'd think to start. Fifty well-chosen cases will catch most of the problems that matter for a narrow workflow, and you can build fifty in an afternoon with two operators. Grow the set over time. Every production failure becomes a new test case, so the same mistake can't ship twice.
Messy fields and conflicting records in your sample are a preview of what the agent will face. Our AI data readiness field note covers how to check the inputs before you blame the agent.
Common mistake: writing test cases that are cleaner than reality. Keep the typos, the forwarded email chains, and the missing attachments. The agent will see all of them.
What to score when you evaluate an agent
Score four things. Most teams only score the first one.
| What you score | The question | How to check it |
|---|---|---|
| Outcome | Did it reach the right result? | Compare to the operator's answer for that case |
| Actions | Did it call the right tools, with the right inputs, in a sane order? | Read the action log against the expected steps |
| Judgment | Did it escalate when it should have, and only then? | Check refuse and hand-off cases separately |
| Cost and speed | Did it finish within a reasonable time and spend? | Track tokens, tool calls, and run time per case |
Outcome
Was the classification right, the reply accurate, the record updated correctly? Structured outputs get a code check. Written replies need a person or a rubric.
Actions
This is where agents surprise people. An agent can land on the right answer by a path you'd never allow: looking up a customer by name instead of ID, retrying a payment call three times, reading a record it shouldn't have access to. Score the path. Your guardrails stop the worst actions in production, but evaluation is where you learn which ones the agent keeps reaching for.
Judgment
Split your pass rate into two numbers: how often it handles cases it should handle, and how often it escalates cases it should escalate. One blended score hides the failure that matters most. An agent at 95% overall that never once escalates a refund over the limit is not a 95% agent.
Cost and speed
Forty tool calls to reach a right answer is still a problem. Long, looping runs usually signal confusion, and they cost real money at volume.
Expert tip: weight your cases. A wrong answer on a low-stakes routing question and a wrong payment are not equal failures. Tag each case low, medium, or high stakes, and require a perfect score on the high-stakes ones before launch.
How to grade: people, code, and model graders
There are three ways to grade an agent's output. Use all three, each where it fits.
Code checks handle anything with a clear right answer. Did it pick category B? Did it call the lookup tool before the update tool? Is the date in the right format? Fast, cheap, and consistent. Use them wherever you can.
People grade what code can't: tone, completeness, whether a reply would satisfy the customer. Use the operators who do this work today. Give them a short rubric, three to five criteria with a clear pass and fail for each, so two graders reach the same verdict.
Model graders use a second AI model to score outputs against your rubric. They scale well and they're useful for written replies once you trust them. But a model grader is itself an agent you have to evaluate. Before relying on one, have a person grade the same fifty cases and check that the grader agrees. If it doesn't, fix the rubric or keep the person.
Our opinion: start with people and code. Add a model grader only when human review becomes the bottleneck. Start with one and you often end up measuring how well two models agree, which tells you nothing about whether the work is right.
The NIST AI Risk Management Framework treats measurement as a core function of trustworthy AI. It's written for organizations, not engineers, and worth a skim before you set launch criteria.
Step-by-step: run your first AI agent evaluation
Here's the sequence we use when we scope an AI agent build. It fits in a week for one narrow workflow.
- Define the job in one paragraph. What the agent does, what it never does, and when it must hand off. If you can't write this, you're not ready to test.
- Pull 50 real cases. Random recent cases, plus the painful ones, plus the ones it should refuse.
- Write the expected result for each. The outcome, the key actions, and whether it should escalate. Operators write this, not the people building the agent.
- Tag the stakes. Low, medium, or high for each case.
- Run the agent on every case and log everything. Inputs, every tool call, the final output, tokens, and time.
- Grade. Code checks first, then people for the rest.
- Set the bar before you look at results. For example: every high-stakes case passes, every must-escalate case escalates, and the overall pass rate clears a level the operators agree to.
- Fix, rerun, compare. Change one thing at a time, rerun the full set, and keep the history.
Step seven is the one teams fudge. If you set the bar after seeing the score, the bar moves to wherever the score landed. Decide first.
Once the agent clears the bar offline, run it in shadow mode: it processes live cases, but a person still does the real work and compares. That's the bridge between evaluation and a real automation pilot.
Keep evaluating after launch
Launch is when evaluation starts paying for itself.
Rerun the full test set on every change. A new model version, a prompt edit, a new tool, a changed API. Treat it like a regression suite in software: nothing ships without passing it. Most model providers let you pin a model version, so upgrades happen when you choose, after the tests pass.
Sample production too. Each week, pull a small random set of live cases and have an operator grade them with the same rubric. Production drifts in ways your test set won't capture: new products, new customer phrasing, seasonal spikes. The weekly sample shows it early.
And route every failure back into the test set. A customer complaint, a reversed action, an escalation that should have happened sooner. Each one becomes a case. Over a few months the test set becomes your record of exactly where the agent breaks.
Who owns this? Name a person. The automation runbook is the right home for the test set, the rubric, and the weekly sampling routine, and the pass rate belongs next to the other automation metrics your ops team tracks.
Common mistakes in AI agent evaluation
- Testing only the happy path. If every case in the set has a clear right answer and complete data, you're testing a demo.
- Letting the builders write the expected answers. They know what the agent can do and grade toward it. Operators know what the work needs.
- Testing once. An evaluation you don't rerun on every change is a snapshot of a system that no longer exists.
- Changing three things at once. If the score moves, you won't know which change moved it.
A real-world shape: insurance verification
Take pre-appointment insurance verification, a workflow we're working on now with a multi-location dental group in healthcare and clinical operations. Before each appointment, someone confirms the patient's coverage and records the benefits.
The evaluation questions show up right away. Did the agent check the right payer for this patient? Did it record coverage that matches what the portal actually said? When the portal was unclear or needed a phone call, did it hand the case to staff with enough context, or did it guess? The painful cases are easy to name: secondary insurance, a recent plan change, a patient whose name doesn't match the payer record.
None of that is about clever prompting. Each one is a test case, and written down with the expected result, they're the bar the agent clears before front-desk staff trust it with a real appointment.
Frequently asked questions
What does AI agent evaluation mean?
AI agent evaluation is testing an agent against a fixed set of realistic cases with known good outcomes, then scoring its final results, the actions it took, and whether it escalated when it should have. Rerunning the same set after every change shows whether the agent got better or worse.
How do you test an AI agent before launch?
Pull around 50 real cases from your own history, including painful ones and cases the agent should refuse. Have operators write the expected result for each. Run the agent, log every action, grade with code checks and people, and set a pass bar before you look at the results. Then run it in shadow mode on live work.
What metrics should I use to evaluate an AI agent?
Track four things: outcome accuracy, correctness of the actions it took, escalation judgment (split into handle rate and escalate rate), and cost and run time per case. Weight cases by stakes and require high-stakes cases to pass outright rather than averaging them in.
Can I use an AI model to grade another AI agent?
Yes, once you've checked it. Have a person grade the same set of cases and confirm the model grader agrees with them. If it doesn't, tighten the rubric or keep human grading. An unchecked model grader can produce a high score that has nothing to do with whether the work is right.
How often should you re-evaluate an AI agent?
Rerun the full test set on every change: new model version, prompt edit, new tool, or changed data source. Separately, have an operator grade a small random sample of live cases every week to catch drift the test set doesn't cover.
What's the difference between evaluation and guardrails?
Guardrails limit what an agent can do in production, such as permissions, spend caps, and approval gates. Evaluation measures what it actually does, before launch and after every change. You need both. Evaluation tells you which guardrails the agent keeps hitting, and guardrails catch the cases evaluation missed.
Final thoughts
If you can't say how your agent performs on the cases that matter, you don't know whether it's ready. You're hoping. Fifty real cases and an honest rubric get you a number your operators actually believe.
Building that test set is part of how we scope agent work. The Executive Study maps the workflow, pulls the painful cases with your team, and sets the launch bar before anything is built. If you've already got an agent and no test set, tell us what it does and we'll help you write the first fifty cases.
