Before commissioning an AI agent, ask what would make its evaluation fail. A score is not the same thing as a release decision, and a release decision is not proof that the intended change happened in the destination system. The useful buying question is whether the evaluation can distinguish an acceptable outcome from a plausible-looking failure in your workflow.
This guide separates a research finding about a disconnected quality gate from the broader choice of how to evaluate an agent. It then proposes acceptance checks to discuss with an implementation team. The examples are hypothetical, not DataBackfill customer results or guarantees of safe operation.
What the ordering-agent study actually measured
Catching One in Five studies an ordering system using randomized generated multi-turn conversations. It reports 2 of 9 systematic patterns captured in one batch, and no operational failures flagged across 100 rounds in another batch where reviewers recorded 23 distinct defects. Patterns, rounds and defects are different counting units; those figures are not an estimate of defects across all production agents.
Crucially, the authors report that raw judge notes often recognized problems, but the operational pass/fail verdict was disconnected from the quality rubric. This is evidence about that evaluation pipeline, not proof that every LLM judge is unable to detect cross-turn failures. The paper also presents proposed improvements that need further testing.
Primary study: Catching One in Five, methods, gate wiring and validity limits
Our earlier version described the corpus as live customer traffic and overstated the findings as general judge blindness. The corrected reading distinguishes what the evaluator observed, how it categorized the observation, and whether the release rule consumed it.
Keep the evaluator, gate and outcome separate
In a proposed implementation, ask the team to show three separate artifacts: the evidence available to the evaluator, the verdict and reason it produced, and the release rule that turns those results into a pass, a block or human review. A dashboard field should not be assumed to control deployment simply because it is displayed next to the release status.
For actions that change a system, define how the resulting state will be checked. A fluent message saying a task completed is not itself the record of that change. Conversely, a failed network response does not establish that nothing happened. The acceptance plan should distinguish success, failure and an uncertain outcome before deciding whether another attempt is appropriate.
A worked acceptance example: an onboarding approval
Suppose a hypothetical workflow reads an onboarding request, checks required fields, drafts an internal task and proposes an access change. The proposal requires approval from an authorized reviewer. The acceptance condition is not merely a friendly final response. It must respect the approval boundary and produce the agreed record of the result.
- Happy path: valid inputs and an authorized approval produce the intended task and expected state, with evidence the reviewer can inspect.
- Missing approval: the assistant may prepare the proposal, but the access-changing action must not execute. Check the destination state, not just the reply.
- Rejected approval: record the rejection and do not quietly choose another route to perform the same action.
- Duplicate request: prevent an unintended second action or explicitly send the uncertain case to an owner.
- Partial failure: a timeout after an action is an unknown outcome until the relevant state is checked; recovery must not blindly repeat it.
- Evaluator failure: an unavailable judge or unparseable verdict follows an explicit review or block policy rather than inheriting a passing default.
These are proposed test cases, not a universal checklist that certifies a system. Expected outcomes depend on the process and permissions. Use non-sensitive fixtures and sandboxed integrations; acceptance testing should not require deliberately making unauthorized changes to a live customer account.
Test that a failing verdict reaches the release decision
Ask for a demonstration with a deliberately failing fixture. Follow its identifier through the evaluator result, the release rule and the deployment decision. Then change it to an acceptable case and verify the difference. This tests the connection between components; it does not measure the evaluator’s ability to discover every unknown failure.
Record the model, prompt or rubric version, workflow configuration and relevant integration versions used in the test. When one changes, decide which cases must be rerun and who can approve exceptions. Keep the reason for an exception beside the result so a future reviewer can distinguish an intentional waiver from an accidentally ignored check.
Use trajectory evidence without assuming a bigger judge is better
The HORIZON paper reports more than 3,100 trajectories across four domains and uses a trajectory-grounded LLM judge validated with human annotation. It supports studying failures over longer task sequences; it is not evidence that all model-based judging is ineffective.
Primary study: HORIZON, The Long-Horizon Task Mirage
For your project, select evidence that matches the acceptance condition. A deterministic comparison may suffice for a required field or resulting database state. Human review may be necessary for an ambiguous business judgment. A model-based assessment can contribute to open-ended response evaluation, but its inputs, rubric and limitations should be explicit. There is no requirement that the evaluator be as complex as the agent.
Measure the right failures, with a defined denominator
Ask what each rate counts: conversations, actions, defects or recurring patterns. If reviewers identify ten defects and the gate flags six of those same defects, the observed recall on that labeled set is six out of ten. That hypothetical arithmetic says nothing about defects reviewers missed or traffic that was not sampled. Do not combine incompatible counting units into a reassuring percentage.
Review false alarms as well as missed failures, and name the owner of unresolved disagreements. Routine samples and selected high-consequence cases answer different questions; report how cases were chosen rather than implying either is a representative production estimate. Keep review access and retention appropriate to the data in the traces.
What to request in an implementation proposal
Request an acceptance plan with named owners, representative fixtures, expected outcomes, release rules and escalation behavior. Separate the cost of creating and maintaining that plan from model usage, integration work and ongoing review. An initial test suite is a starting point for an operating process, not a promise that future failures are eliminated.
Explore AI agent implementation scope and handover
Discuss your workflow and evaluation requirements
Bring a non-sensitive description of the workflow, the actions it may take and examples of unacceptable outcomes. The first decision is what evidence would justify accepting the implementation. Only then does a judge score become useful in context.