A Step Toward Agent Diagnostics
Diagnosing agent failures using user-simulated trajectories?
Two weeks ago, I wrote about the current state of agent evaluations and the gap between research and what product teams actually need.
Most research still focuses on one question:
Did the agent complete the task correctly?
But for teams building agentic products, we also want to know:
“Where did the agent fail?”
“Why did it fail?”
“What should we fix?”
A few days after I published that agent evals article, a new research paper came out that addresses this problem: Counsel, a meta-evaluation dataset for agentic tasks.
As an AI evals practitioner, I do not always find research papers immediately useful. But this one is worth reading.
In case you do not have time for the full paper, here is a quick rundown. ;-)
A quick refresher on τ -bench
Counsel builds on earlier agent-evaluation research, including τ-bench.
τ-bench simulates realistic customer-service users in retail and airline scenarios. An agent must talk with the user, follow policies, and use tools such as database lookups and API updates to complete the request. The benchmark then checks whether the final result is correct.
Later versions expanded the setup. τ2-Bench introduced more complex interactions, while τ3-Bench moved beyond text-only evaluation into multimodal and knowledge-aware agent tasks.
But the core idea remains the same:
Use an LLM to simulate a user, let it interact with an agent under different scenarios, and evaluate whether the agent successfully completes the task.
From pass/fail to diagnostics
τ-bench is useful, but its primary output is still pass or fail.
That tells us whether the agent completed the task, but not necessarily where the failure happened or what caused it.
To improve an agent, we need to zoom in on the specific step, decision, tool call, or policy violation that led to the failure.
That is the diagnostic gap I highlighted in my previous article.
Counsel starts from this gap.
Evaluating the evaluator
Naturally you might think,
Can’t we just use an LLM to run diagnosis just like what we do in LLM-as-a-judge?
Instead of evaluating only the final outcome, can we have the judge inspect each step of the agent’s trajectory and ask:
Did the agent make an error here? If so, what was the error?
I suspect many of us have already tried this, myself included, when we got tired of manually review traces.
But it has not become standard practice, partly because LLMs have not been consistently reliable at analyzing long agent traces.
So the real question becomes:
How do we know whether the judge’s diagnosis is correct?
This is why Counsel is a meta-evaluation dataset. Instead of evaluating the agent, it focuses on evaluating the evaluator that critiques the agent.
Methodology + Findings
The idea of the research is straight-forward.
The researchers generated agent trajectories using two existing user-simulators, one of which is τ-bench.
For each agent trajectory (from agent and user simulator interactions), the researchers asked open-weight LLM judges (GPT-OSS and Qwen3) to evaluate the agent’s actions one step at a time.
At each agent-generated step, the judge received the conversation and tool context available up to that point, without seeing what happened later.
The judge then produced two outputs for each step:
Whether the step contained an error
A explanation of what went wrong
Human experts (data scientists) then reviewed the errors flagged by the judges and determine if they are
Spot on, or
Correct location, poor reasoning, or
Should not have flagged
The analysis was done based on human labels.
The most interesting result is the gap between error localization and diagnostic reasoning. The strongest judge reached roughly:
88% agreement with humans on the error location
65% agreement on the explanation
The judge was relatively good at identifying which step looked problematic. But it was much less reliable at explaining why that step was wrong.
It’s also shown that stronger judge models generally produced critiques that agreed more closely with human reviewers; and giving a judge model more reasoning effort also improved performance. This is not particularly surprising, but it is always good to see quantitative results that confirm our guesses.
My two cents
Counsel is a meaningful step toward more automated agent diagnostics.
It gives us a way to quantify how accurately an LLM judge can diagnose agent traces, rather than simply assuming the critique is correct.
But there is still a lot of work to do.
The process of “evaluating the evaluator,” or meta-evaluation, remains tedious and slow. Agent trajectories can vary significantly, and reviewing them still requires human experts.
Counsel also focuses mainly on errors that the judge already flagged. It tells us how often those critiques are correct, but not necessarily how many failures the judge missed entirely.
Hopefully, we will see more meta-evaluation research and more judge models specialized in agent diagnostics.
When that day comes, agents may start improving themselves based on feedback from automated diagnostics—and we may all lose our jobs.
Until then, at least there will still be plenty of trace-labeling work for humans.
Just kidding.
If you’d like to go deeper into AI evals, I teach a popular Maven course AI Evals & Analytics Playbook.
The course is fully self-paced, so you can work through it on your own schedule. We also offer one-on-one office hours for more personalized support.
We currently have one open cohort and another starting soon. The current cohort began on June 29, and you are still very welcome to join! The next cohort begins on July 20.
As a Data Science x AI reader, you can use this link for 30% off with promo code DSxAI.
https://maven.com/ai-evals-and-analytics/ai-evals-analytics-playbook?promoCode=DSxAI
