Evals: how to measure whether an agent does its job well before your customer notices
Evals for AI agents in production: define tasks, score results, analyze failures and improve with data before a single error ever reaches your customers.
The problemYou change something in your agent and don't know whether you improved it or broke it until a customer complains.
Monday, 8:05 a.m. Marta, customer service manager at an online auto parts retailer, finds eleven new complaints in the inbox. They all say the same thing: the assistant denied a return that should have been approved. The previous Thursday, the team had tweaked the agent's instructions so it would sound "friendlier." The tone improved. The return policy, without anyone noticing, stopped being applied correctly. The change was not the problem. The problem was not having evals for AI agents to catch it before it went live.
Anyone who has put an agent into production knows the feeling: every tweak is a gamble. This article explains how to stop gambling and start measuring.
Why "I tested it and it worked" is not enough
When a team adjusts an agent, they usually test it with three or four questions, see that it answers well and ship it. The problem is that an agent handles hundreds of different situations. A change that improves one case can make another worse, one that nobody checked.
On top of that, the agent changes even if you don't touch it. Providers update their models, business data shifts, new products get added. Without a systematic way to measure, you only find out something broke when a customer tells you.
That is why so many pilots stay pilots. A March 2026 survey of 650 technology leaders found that 78% have at least one agent pilot, but only 14% have scaled it across the organization. Without evidence of quality, nobody dares to scale; we dig into this in why your AI pilot never made it to production.
What evals for AI agents are, in plain terms
An eval is an exam. It has questions (real tasks the agent must handle), a way to grade them (what counts as good, fair or poor) and a result you can compare over time.
Think about how an airline pilot gets certified. Flying well once is not enough. They go through a simulator with defined scenarios, including the hard ones: a storm, an engine failure, a wet runway. If the aircraft changes one day, they go back to the simulator. Evals are your agent's flight simulator.
The improvement cycle in five steps
Evals are not a one-time audit. They are the engine of the agent's improvement cycle.
1. Define the tasks
Gather real cases that represent the agent's work. In Marta's case: return requests inside and outside the return window, products under warranty, upset customers, ambiguous questions. Include the hard cases and the ones that have failed before. This set is called a benchmark: your fixed battery of tests.
2. Run
Run the agent against the entire benchmark, under the same conditions it works in production: same tools, same rules, same business memory.
3. Score
Each response is compared with what was expected. Some things can be checked automatically (did it apply the right policy? did it cite the order number?). Others require judgment, and that is where an independent evaluator agent can step in, working from rules your team wrote. The golden rule: whoever generates the work does not grade it, as we explain in generator and evaluator agents.
4. Analyze the failures
This is where the real value is. Knowing that the agent got a certain share of cases right is not enough. You need to group the errors: is it failing for lack of information? Because of an ambiguous instruction? Because a tool returns incomplete data? Because of the model?
5. Improve and measure again
Depending on the cause, the fix is different:
- Instructions: clarify a rule or add an example in the prompt.
- Tools: let the agent look up the actual purchase date instead of asking for it.
- Policies: add a hard rule, for example "never deny a return without checking the delivery date."
- Model: switch to a different model for a specific step if the current one does not deliver the required level.
After every fix, you rerun the full benchmark. If the result improves and nothing breaks, you ship. If something gets worse, you know before your customer does.
A practical framework: what to measure by process
| Process | What counts as success | What counts as a serious error |
|---|---|---|
| Customer service | Resolves the case applying the right policy | Promises something the company won't deliver |
| Sales proposals | Current prices and terms | Unauthorized discount |
| Contract review | Flags the defined risk clauses | Misses a critical clause |
| Lead qualification | Classifies according to the team's criteria | Discards a qualified lead |
Look at the third column. A good benchmark does not treat all errors the same. A tone error does not weigh as much as a false promise.
Illustrative example: the returns assistant, before and after
Illustrative example. Before, Marta's team adjusted the agent every week and tested it by eye. Problems showed up as complaints.
Then they built a benchmark of 60 real cases from recent months, including the eleven that failed that Monday. Every change runs through the benchmark before going live. The day someone tweaks the tone again, the eval immediately flags that returns inside the window have started failing. The change does not ship until it is fixed.
What is interesting is that failure data feeds the next cycle. Every real run flagged as problematic in the traces becomes a new case in the benchmark. Without observability, you would have nowhere to get those cases from.
Common mistakes
- A benchmark with only easy cases. If everything always passes, the test is not measuring anything.
- Letting only the technical team design it. Quality criteria belong to the business.
- Measuring once. An eval that is not repeated after every change is a snapshot, not a control.
- Looking only at the average. An agent can improve on average and get worse precisely on the cases that matter most.
- Never updating the benchmark. Your business changes; your tests should change too.
What to do on Monday
Ask the person who owns the process to gather twenty real cases the agent should handle, including five hard ones and the errors you already know about. For each one, have them write a single line describing what a correct answer would be. That list is your first benchmark.
At Addabra, evals are part of the method, not an add-on: we define the benchmark with your team during design, run it before every change and use the failures to improve instructions, tools, policies or the model. That way, every version of the system reaches production with evidence that it works.
Frequently asked questions
What is an eval for an AI agent?
It is a controlled test: you give the agent a task with a known expected answer and measure how well it handles it. A set of evals works like the exam the agent must pass before every significant change.
Who decides what a good answer looks like?
The people who know the business, not the technical team. The customer service manager, the attorney or the head of sales describe what result is acceptable, what counts as a serious error and what is tolerable. The technical team turns those criteria into automated tests.
How often should evals run?
Whenever something that affects the agent changes: its instructions, its tools, its rules or the model it uses. It also makes sense to run them on a regular schedule, because providers' models change too and behavior can shift without you touching anything.
