How to Evaluate AI Agents: A Test Plan for Production

Most AI agents demo beautifully and break in production. The gap is not the model. It is that the team never built a way to measure whether the agent actually does the job before real users depend on it. A polished demo proves the agent can succeed once, on a path you chose. Evaluation proves it succeeds repeatedly, on paths your customers choose, including the ones that go wrong.

Evaluating an agent is different from evaluating a chatbot. A chatbot returns text you can eyeball. An agent takes actions: it calls tools, reads and writes to your systems, chains several steps, and sometimes spends money or sends an email on a customer's behalf. You are not grading a sentence. You are grading a sequence of decisions with side effects. That changes what you measure and how.

This is a field guide to testing agents before they ship: what to build, what to score, and what gate to put in front of production.

Start with an eval set, not a vibe

The first artifact you need is not a dashboard. It is a set of test cases, concrete inputs paired with what a correct outcome looks like. Treat it like a test suite for behavior.

Build it from three sources:

• Real traffic. Pull 50 to 200 actual requests from logs, support tickets, or the manual process the agent replaces. This is your ground truth for what users genuinely ask.
• Known failure modes. Every domain has them, ambiguous requests, missing fields, two valid interpretations, requests the agent should refuse. Write these down on purpose.
• Edge cases that cost money. A refund agent that approves a $40,000 refund, a scheduling agent that double-books a surgeon, a data agent that runs an unbounded query. The rare expensive failures matter more than the common cheap ones.

For each case, define what "good" means before you run anything. Sometimes that is an exact answer. More often it is a rubric: did the agent gather the right information, take an allowed action, and stop when it should. Write the rubric down. An eval set you can rerun on every change is the difference between engineering and guessing.

A useful rule of thumb: if you cannot describe a passing result in one sentence, you do not understand the task well enough to ship an agent for it.

Score the trajectory, not just the final answer

The most common evaluation mistake is grading only the last message. An agent can give the right answer for the wrong reasons, and the wrong reasons are what bite you at scale.

Score the whole trajectory:

• Outcome: did it reach the correct end state? Right refund amount, right meeting booked, right ticket routed.
• Tool use: did it call the right tools with the right arguments? An agent that guesses an order ID instead of looking it up got lucky, not correct.
• Path efficiency: how many steps, how many tokens, how much latency? An agent that takes 14 tool calls to do a 3-call job is a cost and reliability problem even when the answer is right.
• Recovery: when a tool returned an error or empty result, did the agent adapt or hallucinate around it?
• Stopping behavior: did it know when it was done, and did it refuse or escalate when it should have?

Capture full traces, every prompt, tool call, argument, and response, for each run. When a case fails, the trace tells you whether the model reasoned wrong, the tool returned bad data, or the prompt was ambiguous. Without traces you are debugging blind.

Combine automated graders with human review

You cannot hand-review thousands of runs on every change, and you cannot fully trust an automated grader either. Use both, deliberately.

For checkable outcomes, write code assertions: the refund equals the invoice, the JSON validates, the database row exists. These are cheap, deterministic, and run on every commit.

For judgment calls, was the tone appropriate, was the summary faithful, did the answer actually address the question, an LLM-as-judge works well, but only if you calibrate it. Have a human grade 50 cases, then check that the judge agrees with the human at a rate you trust. An uncalibrated judge gives you confident, repeatable, wrong scores.

Keep humans in the loop for the cases that matter most: anything irreversible, anything customer-facing, anything where the cost of being wrong is high. The point of automation is to spend scarce human attention on the 5% of cases where it changes a decision, not the 95% a script can settle.

Test the failure modes that actually hurt

Happy-path accuracy is table stakes. The agents that survive production are the ones tested against adversarial and degraded conditions before launch:

• Prompt injection: a document, email, or web page that tells the agent to ignore its instructions. If your agent reads untrusted content, this is a security test, not a nice-to-have.
• Tool failures: make the API time out, return a 500, or return plausible-but-wrong data. The agent should degrade gracefully, not invent a result.
• Ambiguity and missing data: does it ask a clarifying question, or does it confidently guess?
• Scope creep: does it attempt actions outside its mandate when a user asks nicely?
• Cost blowups: the runaway loop, the unbounded retry, the query with no limit.

Run these as a standing suite. Each one you discover in production should become a permanent regression case so it never ships again.

Wire evaluation into the deploy pipeline

Evaluation that runs once before launch is a one-time photograph. Production agents drift, models get updated, prompts get tweaked, tools change behind the same interface, and your own data shifts underneath the agent. The teams that keep agents reliable run their eval set on every prompt change, every model swap, and every tool update, and they block the deploy when scores regress.

This is exactly the part most teams underestimate, and it is where moving from a working prototype to a system that earns trust in real workflows gets hard. Taking agents from a clever demo to something that runs reliably inside your actual stack, with the offline eval suites, online monitoring, and rollback gates that keep them honest, is the core of what it means to deploy AI agents into production rather than just build them.

The mechanics that matter:

• A regression gate in CI that fails the build if accuracy on the eval set drops below threshold.
• Online metrics in production, task success, escalation rate, tool-error rate, cost per task, sampled and reviewed continuously.
• A feedback loop where production failures flow back into the eval set, so the suite grows sharper as the agent ages.
• A rollback plan that does not require a heroics-and-pager-duty night.

Pick one number that means "this agent is doing its job", resolved tickets without escalation, correctly processed invoices, whatever your business actually cares about, and track it from day one. Vanity metrics like "responses generated" tell you nothing about whether the agent works.

What good looks like

You know your evaluation is mature when three things are true. You can answer "is this version better than the last one?" with a number, not an opinion. A teammate can change a prompt and see within minutes whether they broke anything. And when an agent fails in production, you can reproduce the failure in your eval set, fix it, and prove the fix holds.

Get there and shipping stops being a leap of faith. You launch because the agent cleared a bar you defined, and you keep it healthy because the same bar runs every day after. That is the whole game: evaluation is not a gate you pass once before launch, it is the instrument panel you fly the agent with for as long as it runs.