Your Eval Gate Is a Rubber Stamp: Why LLM-as-Judge Misses Most Long-Horizon Agent Failures

Your CI pipeline says the agent passed. Your judge gave it a 9 out of 10. Your dashboard is green. And your users are still hitting the same broken conversation three days a week. This is not a tuning problem. It is an architecture problem, and the last few weeks of research have made the shape of it uncomfortably clear.

The number that should worry you

A recently published study on a deployed multi-turn food-and-beverage ordering agent ran the agent's own LLM judge against human review batches on live production traffic. The results were not close. Across three review batches, the judge caught 2 of 9 human-confirmed systematic problem patterns in one batch, roughly 22 percent. In another batch, the judge's operational gate flagged zero out of 100 rounds as problematic, while human reviewers independently confirmed 23 distinct defects in that same batch. Zero and twenty-three are not a calibration gap. That's a judge that never had a chance to see the problem in the first place.

This matters because that judge was not obviously broken. It had been validated the way most teams validate judges: by checking agreement with human ratings on a held-out sample. It passed that check. It still missed the majority of real defects once it was running as an automated gate on live traffic. The validation methodology and the deployment methodology were measuring two different things, and almost nobody was measuring the second one.

Agreement is not recall

Most eval reports answer the question 'does the judge agree with a human on this sample of outputs.' That's a useful number, but it's the wrong number if the judge is going to sit in your deploy pipeline deciding what ships. The question that actually matters for shipping software is different: of the real problems present in production traffic, what fraction does the automated gate actually catch? Agreement on a curated sample and recall of real-world defects diverge fast, especially once the traffic gets messy, multi-turn, and full of the weird edge cases production always generates that your eval set never anticipated.

A judge that agrees with humans 90 percent of the time on a benchmark can still miss the majority of real defects once it's grading live, multi-turn traffic instead of a curated sample.

Teams keep conflating these two numbers because agreement is the number that's easy to compute and easy to report. Recall against unknown production failure modes requires a human review process running in parallel to the judge, comparing its verdicts against what actually happened, and that's expensive and slow. So teams skip it, ship the judge as a gate, and find out about the gap the same way the food-and-beverage team did: months later, via an angry ops review, after the judge has been rubber-stamping broken conversations the whole time.

The blind spot is structural, not random

The most important finding in the research isn't the specific percentages. It's the shape of the failure. Researchers found that the judge's blind spot generalizes across a consistent pattern: cross-turn failures are missed far more often than turn-local ones, the rubric axes the judge grades against don't line up with the axes that actually determine shipping quality, and the operational gate ends up disconnected from the rubric it's supposedly enforcing.

LLM judges are good at one specific task: looking at a single response in isolation and asking whether it's a reasonable answer to the immediate prompt. That's exactly what they were trained and prompted to do, and it's exactly what most eval harnesses ask them to do. Long-horizon agent failures rarely look like a single bad response. They look like state quietly drifting across turns. An order gets modified in turn 4, the agent forgets the modification by turn 9, contradicts itself in turn 12, and the user gives up in turn 14. Every individual turn, read on its own, looks fine. A turn-local judge scores all fourteen of them as reasonable. The failure only exists at the level of the trajectory, and a judge that never reads the trajectory as a whole will never see it.

Rubrics rot faster than you think

Even when a judge is architecturally capable of catching a failure class, it needs a rubric that describes what to look for. Rubrics are usually written once, against the agent's behavior at the time, and then left alone. That's the second half of the problem. Changes to the underlying model, the tool set, or the way users actually interact with the agent all shift its behavior. A fixed rubric keeps grading against criteria that described last quarter's agent. It doesn't know the tool schema changed, that a new user workflow emerged, or that the model swap altered how the agent handles ambiguous requests. The judge keeps returning high scores because it's still checking for the old failure modes while a new one grows in the blind spot underneath it.

This is why the framing among evaluation-tooling teams has shifted from 'grade the final answer' to 'investigate the entire trajectory.' Grading a final answer is a snapshot. Investigating a trajectory means looking at what the agent saw at each step, what it decided to do, what state it changed, and whether that sequence of decisions holds together, not just whether the last message reads well.

What the long-horizon benchmarks confirm

This isn't isolated to one production system. A large-scale academic study running over 3,100 agent trajectories across multiple model families found that long-horizon breakdown is not simply a gradual drop in task success rate as tasks get longer. It's a structural shift in the composition of failures. The kinds of things that break at turn 5 are different from the kinds of things that break at turn 50. Early failures tend to be local: a bad tool call, a misread instruction. Late failures tend to be compounding: accumulated state drift, contradictions with earlier commitments, loss of the original goal under the weight of context that's grown too large to track cleanly.

The practical consequence is that a static eval suite tuned on short tasks will systematically miss what breaks in long-running production agents, no matter how well-calibrated it was when you built it. If your eval set is full of five-turn transcripts and your production agent runs for forty turns, you are testing a different failure regime than the one your users experience.

What actually works: milestones over verdicts

The response taking shape across both research and vendor tooling is not 'build a bigger judge' or 'write a better prompt for the judge.' It's a shift in how evaluation itself is structured. A few patterns are converging:

• Break long-horizon tasks into phase-level checkpoints with partial credit, so a trajectory that fails at turn 30 still gets credit and diagnosis for what happened correctly in turns 1 through 29, instead of one binary pass/fail verdict on the whole run.
• Trace at the trajectory level by default, not the turn level. Store what the agent saw, what tools it called, what state it changed, and what it decided, so a reviewer or a second-pass judge can reconstruct the whole arc instead of guessing from a final transcript.
• Decouple rubric maintenance from model and tool releases. Every model swap, tool schema change, or new user workflow should trigger a rubric review, not a silent continuation of the old grading criteria.
• Run human review in parallel with the automated gate on a rotating sample of live traffic, and measure recall of confirmed defects, not just agreement with a curated benchmark.
• Treat the eval harness as an agentic system in its own right: something that searches, verifies, and adapts across a run, rather than a single prompt that issues one verdict at the end.

That last point is the biggest mental shift. Most teams built their judge the same way they built their first agent prototype: one prompt, one model call, one score. The agents being evaluated have since become multi-step, stateful, tool-using systems. The judge grading them is often still a single LLM call reading a flattened transcript. You cannot evaluate a system with more moving parts than your evaluator using an evaluator with fewer moving parts. The judge needs at least as much structure as the thing it's judging: the ability to inspect intermediate state, check specific claims against tool outputs, and flag when a rubric axis doesn't apply to the trajectory in front of it.

The uncomfortable rebuild

None of this is an argument against LLM-as-judge. It's an argument against treating a single-pass judge validated on agreement scores as a sufficient gate for production traffic that has already outgrown the assumptions the judge was built under. The teams getting burned aren't the ones without an eval process. They're the ones with an eval process that looked rigorous on the day it shipped and quietly stopped matching reality every time the agent, the model, or the users changed underneath it. An eval gate that isn't revisited on the same cadence as the agent it grades isn't a safety net. It's a green checkmark generator, and the gap between what it reports and what's actually happening in production only shows up after someone goes looking for it by hand.