A Practical Guide to Agent Evals: Don’t Just Look at What It Says—Look at What the World Becomes
An Agent says, “Your flight has been booked,” but there is no reservation in the database. The first suspect is obvious: the Agent finished talking, but it did not finish the job.
But the case is not that simple. In another flight-booking task, Opus 4.5 found a valid, policy-compliant solution that the test did not anticipate, yet the rigid grading rules still marked it as a failure. This time, the Agent was not lying. The measuring stick was broken.
These two failures point in opposite directions: one mistakes failure for success, while the other mistakes success for failure. The trouble with evaluating Agents lies somewhere between them.
We have already covered graders and nondeterministic metrics in GP-151, final-state verification in GP-201, and how to turn production failures into the next round of tests in GP-158. Here, we go straight to the acceptance test: how do you catch false completion without rejecting valid solutions?
Scene One: The Response Says Success, but the World Does Not
Agents call tools across multiple turns and modify their environments, and an error in one step can snowball through everything that follows. Reading only the final response is like collecting nothing but a statement after the crime.
Every trial run should leave behind at least two kinds of evidence: an execution trace that preserves outputs, tool calls, and intermediate results, and a final state that records what the environment ultimately became. The first answers, “What happened during the process?” The second answers, “Did the job actually get done?” You also need to run the same task multiple times to see the variation in model output.
Objective outcomes can be checked by code, while semantic and interaction quality can be judged by a model. Open-ended tasks can then be calibrated through human spot checks. You do not need one omnipotent judge; you need different judges looking at the evidence each is best equipped to assess.
Mogu inner monologue:
Reading only the Agent’s final “Done” is like seeing “Delivered” in a food delivery app but never opening the door to check whether your meal is there. The execution trace can explain the process; only the final state can verify whether the job was truly completed. GP-201 is dedicated to this kind of acceptance-test design.
The evaluation framework running these tasks provides tools, executes trials in parallel, preserves traces, and aggregates scores. The Agent Harness, meanwhile, enables the model to call tools and complete the work. Claude Code is one kind of Agent Harness. So the score measures the model and its operating framework as a whole, not just the model alone.
The same model can perform very differently when given different tools, prompts, retry strategies, and state management. If the eval environment does not reproduce the combination used in production, then it is measuring a different system. When comparing models, you also need to record differences in the Harness.
Different tasks leave behind different kinds of “crime scenes.” For coding tasks, you need to see whether the tests pass, but you also need to read the execution trace so the Agent does not fix the error while leaving behind an unmaintainable mess. In customer service, you need to verify states such as refunds and reservations, while also looking at the number of turns and the tone. If necessary, another model can simulate the user. Getting the job done, getting it done within ten turns, and getting it done without infuriating the user are three different signals.
Research Agents do not have unit tests that can directly declare success or failure. You need to separately check whether claims are sourced, whether all necessary facts are present, and whether the sources are reliable. The longer the answer and the more open-ended the question, the easier it is for errors to hide inside paragraphs that look complete.
One research benchmark frames its questions as “easy to verify, hard to solve”: once you find the answer, it is clear whether it is correct, but finding it is like searching for a needle in the haystack of the open web. That is close to many real-world research tasks.
Computer-use Agents need both the screen and the backend state checked: seeing a confirmation page does not mean the order was actually placed. They also need to trade off speed, latency, and Token usage when choosing between web structure and screenshot-based interaction. These four task categories may look different, but they are all pursuing the same question: which evidence comes closest to real success?
Scene Two: The Agent Was Right, but the Measuring Stick Was Broken
Finding the right evidence solves only half the problem. The next trap is that the grader may inspect the wrong evidence, rely on assumptions never stated in the task, or even blame the Agent for infrastructure failures.
Every trial run must start from a clean environment. Leftover files, caches, or exhausted resources can contaminate the result; shared state may even slip answers to the Agent. Anthropic observed Claude gaining an unfair advantage from git history left behind by an earlier trial run.
This case of “peeking at the previous trial through git history” is painfully real. If the environment is not cleanly isolated, the Agent may accidentally “learn” information it should not know. The score may look great, but it becomes completely untrustworthy.
If several trial runs all fail because of the same memory bottleneck, they are not independent samples either. The environment must resemble production without importing production’s dirty state along with it; otherwise, the score is measuring infrastructure noise.
Graders should verify outcomes whenever possible instead of forcing the Agent to call tools in a prescribed order. If the route is hard-coded too rigidly, the grader will reject valid solutions it did not anticipate. Composite tasks also need partial credit: completing identity verification but failing to issue the refund is still closer to success than getting stuck at the beginning. Breaking the task into meaningful checkpoints also makes it easier to see whether you should fix the model, prompt, tools, or grader.
Model judges need to be calibrated against human expert judgment, and they should be allowed to answer “Unknown” when there is not enough information. Compared with asking one model for a vague overall score, defining clear rules for each dimension and judging them separately makes it easier to locate the failure.
“Give the model a way out” is a very practical move. If you force a judge to assign a score even when the information is insufficient, it will invent a justification. Letting it say “Unknown” produces a more honest signal.
The most absurd moments usually come down to decimal points. Among the cases Anthropic collected, one grader marked “96.12” wrong because it expected “96.124991…,” while another task required reaching a threshold but the grader accepted only values above it. A fair failure should make it immediately obvious what the Agent did wrong, not force people to guess which hidden rule the grader was enforcing.
After one benchmark fixed precision comparisons, ambiguous specifications, and irreproducible stochastic tasks, Opus 4.5’s score rose from 42% to 95%. The model did not become smarter overnight; the measuring stick was repaired. When a score is extremely low or suddenly skyrockets, the Agent may not be the first thing you should inspect.
Graders still need to prevent Agents from gaming the system for points, but that cannot mean locking out every valid path. A “pass” should mean the problem was solved. A “failure” should point to something that can be fixed next.
Don’t Build a Giant Test Suite First—Start by Collecting Wounds
Many teams are intimidated before they even begin by the thought that they need hundreds of tasks. An early-stage system does not need an entire examination authority: 20–50 simple tasks drawn from real failures are enough to get started. Manual checks, must-test behaviors before release, error trackers, and customer support queues are all ready-made sources. Prioritize them by user impact and turn the most painful failures into permanent tests.
When a product is still young, requirements can double as success criteria. After a system has been live for a while, the team may instead need to reverse-engineer what “success” actually looks like from existing behavior. Mature products need more—and harder—tasks to reveal small differences, but the starting point is still a small set of high-value cases.
Every production failure should become an eval task so the same problem does not happen again. Treat failures as fuel for evaluation, and the eval system will grow on its own. GP-158 covers exactly this improvement loop.
A good task should lead two domain experts reviewing it independently to the same pass-or-fail verdict. Any file paths, constraints, and success criteria the grader will check must be written into the task beforehand. Otherwise, the Agent may follow the instructions correctly and still lose to a hidden assumption in the test.
For example, a task might ask for a script without specifying where to save it, while the test looks for the file only at one fixed path. The Agent did not violate the instructions, yet it still receives a zero. This kind of failure measures a specification gap, not Agent capability.
Every task should also include a reference solution known to pass all checks, proving that the task is actually solvable. If a frontier model fails all 100 runs, suspect the task or the grader before rushing to declare the Agent incapable.
The task set should test both what the Agent should do and what it should not do. If you only test whether it searches when a search is needed, you may end up training an Agent that looks everything up. When Anthropic calibrated web search for Claude.ai, it included both questions that should trigger a weather search and questions that could be answered from existing knowledge.
This balance applies to more than search. If you test only customer service cases where refunds should be issued, the Agent may refund every order. If you test only safety refusals, it may refuse to do anything. Positive and negative cases need to appear in pairs so the system cannot inflate its score with a single conservative strategy.
Task sets also tend to have two different destinies. Capability evals deliberately target things the Agent cannot yet do, so a low initial score leaves room for improvement. Regression evals protect things it can already do; a falling score means a change broke something. Once a capability task is solved reliably, it graduates into a regression task and continues guarding against old failures returning.
A popular coding capability benchmark climbed from 30% to over 80% in early 2026 and was already approaching saturation. When the test has nothing left but top scorers, it can still guard against regressions, but it can no longer measure much improvement.
Model behavior varies from run to run, and a single pass rate can tell the wrong story. pass@k asks, “What is the probability of succeeding at least once in k runs?” It is suitable for tasks where you only need to find one workable solution. pass^k asks, “What is the probability of succeeding in every one of k consecutive runs?” It fits products such as customer service, where consistent reliability matters. The two move in opposite directions as k increases; GP-151 provides the full derivation.
For coding, finding the right solution on the first attempt is usually most valuable. Research tasks may allow several tries as long as one reliable answer emerges. User-facing actions often need to succeed every time. Publishing only one average pass rate hides the dimension the product actually depends on.
The Hard Part Begins After You Reach a Perfect Score
Once the task set is built and the scores are in, the work is only half done. Teams still need to routinely sample and read execution traces to distinguish genuine Agent errors from valid solutions rejected by the grader. The details flattened by an average score survive only in those traces.
“Read the execution traces” sounds like basic advice, but too many teams make decisions from the final score alone. A score is compressed information—if you do not know what happened behind it, you cannot know whether to trust it or question it.
Once a task set approaches a perfect score, its remaining value is guarding against regressions, and the team needs to add harder new tasks. A high score does not always mean the product is finally perfect; sometimes the test has simply grown old.
Task sets also need clear owners: the eval team maintains the infrastructure, while domain experts and product teams keep adding real-world tasks. This work should be as routine as maintaining unit tests.
An ownerless task set slowly decays. Product behavior changes, but old success criteria remain. Reference answers go stale, yet the judge keeps deducting points against them. Maintenance is not about occasionally adding a few tasks; it means continuously checking that every failure is still fair and every pass is still meaningful.
Capability evals can define in advance what models cannot do today but the product hopes they can do a few months from now. When a new model arrives, rerunning the task set reveals which bets paid off. Saturated old tasks can continue catching regressions without being forced to prove that capability improved yet again.
Automated evals catch mistakes quickly before launch. After launch, production monitoring catches distribution drift and unexpected situations, user feedback fills holes the task set never anticipated, and—once there is enough traffic—controlled experiments can confirm whether a change actually helps users. Human reviewers do not need to take over every task, but they must keep calibrating judges that depend on subjective judgment.
Conclusion
You do not need a perfect task set to begin. Start by turning 20–50 real failures into clear, solvable, isolated tasks. Then investigate two things: did the Agent actually change the world correctly, and did the measuring stick judge the Agent correctly?
When an Agent says, “Your flight has been booked,” do not rush to issue the boarding pass. Check the database—and the measuring stick in your hand. ٩(◕‿◕｡)۶