What About Agent Evals

When do we move from pass/fail to where and what to fix?

AI evals are always a couple steps behind AI product development.

First, we had model benchmarks, then RAG eval metrics, then text-only evals were not enough and we need to look into multimodal evals. And of course, multi-turn evals since single-turn performance often does not predict real-world user experience.

Now, just as we are getting better at evaluating multi-turn interactions, GenAI products have moved again to agents.

So once again, we need to catch up.

Why is Agent Evals hard?

So why is agent eval different?

In practice, evals (agentic or not) usually serve two goals.

• Release decision-making: does the AI product meet expectations, and is it ready to ship?

• Diagnostics: when something goes wrong, can we help the development team understand what failed and what to fix?

Agent evals inherit both goals, but each becomes harder.

The Challenge of Simulating “env”

For release decisions, the challenge is simulation. Agentic applications are usually designed to solve complex tasks, not isolated prompts. The user interaction is rarely single-turn. It may involve tools, external systems, documents, memory, images, changing requirements, and ambiguous user intent.

So the eval problem is no longer:

“Given this prompt, is the answer good?”

It becomes:

“Can we simulate a realistic task environment and evaluate whether the agent completed the user’s goal?”

In a way this is similar to the concept of constructing an “env” in Reinforcement Learning, which has also been a key challenge in training RL models.

"Where it fails?”

For diagnostics, the challenge is attribution. Agentic systems have many moving pieces, and the workflow is often dynamic rather than a fixed sequence of steps.
When something goes wrong, a bad final answer does not tell us where the system failed.

Was it a planning issue?
A tool selection issue?
A retrieval problem?
A memory issue?
A state management bug?
…

And because each agentic system is different, it is very difficult to generalize and propose a universal solution to root cause analysis.

Agent Eval Frameworks/Benchmarks

Because of these challenges, many current agent evaluation benchmarks share two patterns.

First, they are often end-to-end outcome evaluations.
The agent is evaluated as a complete system in a task environment, and success is often judged by final task completion, final output, or final environment state. This is useful for release-style pass/fail decisions, but it does not always explain why the agent failed.

Second, they are usually scenario- or domain-specific.
Many benchmarks are built around concrete task environments, such as coding, web navigation, customer support, airline booking, or tool-use workflows. This makes the benchmark more realistic, but also harder to generalize across products.

Example: τ-bench

A good example is τ-bench (and τ2-Bench), a benchmark for evaluating tool-using agents in realistic customer-service scenarios such as retail and airline support.

τ-bench is useful because it turns agent evaluation into a simulated task environment. The user is simulated by an LLM, so the agent has to interact over multiple turns instead of receiving a fully specified instruction upfront.

The agent is also given access to domain-specific tools, such as APIs for retrieving orders, updating customer records, or modifying bookings, plus a set of policies it must follow. During a session, the agent needs to ask for missing information, choose the right tools, call them with the right parameters, follow business rules, and eventually complete the task.

At the end, τ-bench evaluates whether the final database state matches the expected goal state, rather than simply judging whether the final response sounds good. It also introduces pass^k to measure reliability across repeated trials, because agent behavior can be inconsistent even on the same task.

Conceptually, the framework may look something like this:

Beyond Pass/Fail

The core idea is that the benchmark is evaluating whether the agent(s) can operate inside a workflow: interact with a user, follow policies, call tools, update state, and complete the task.

This is a meaningful step forward from evaluating model generation.

But for product teams, “the final answer is wrong” is not enough. We also need to know where the agent got lost, and therefore know what to fix it. Or even a step further, why it fails at where it fails, and therefore know how to fix it.

This is the same problem we have discussed with simpler LLM systems. With diagnostic information, development teams are left scratching their heads, repeatedly changing prompts, tools, or workflows through trial and error.

The development process becomes random, even though we are supposedly building intelligent systems.

Agent evals need to move beyond pass/fail scores, and move toward diagnostics.