Why grading one response at a time can’t catch the failures that matter — and what evaluating the whole session gives you instead. Second in a series on building trustworthy AI agents. session-native evaluation actually works — and why grading one response at a time can’t catch the failures that matter. Second in a series on building trustworthy AI agents.
In the first post we made a claim and then walked past it: that to catch how agents really fail, you have to evaluate the whole session, not one response at a time. We called it session-native, said it was the idea Turn3 is built on, and moved on.
This post is the part we walked past. What does session-native actually mean, mechanically — and why is it the difference between a tool that tells you your agent looks fine and one that tells you whether it did its job?
The unit of failure is the session, not the response
Let’s start with a question that sounds simple and isn’t: what is the thing you’re actually evaluating?
For a classic model API, the unit is obvious — one request, one response, grade it. That framing quietly carried over into a lot of LLM tooling: score each completion, average the scores, call it quality.
But an agent doesn’t do one thing. It has a conversation, or runs a multi-step task, across many turns — reading context, calling tools, spawning sub-agents, carrying state forward. The thing you actually care about isn’t “was response #5 good.” It’s “did this whole interaction accomplish what the user needed.”
Those are different questions, and the gap between them is where agent failures live. Consider what a per-response evaluator literally cannot see:
- Contradiction across turns. The agent states something at turn 6 that flatly contradicts what the user established at turn 3. Response #6, read alone, is fluent and plausible. It’s only wrong relative to turn 3 — which the per-response check never looked at.
- Goal drift. The user asked for one thing. Twelve turns of individually-helpful responses later, the agent is confidently solving a subtly different problem. No single response is bad. The trajectory is.
- Dropped constraints. “Don’t recommend anything over $50.” Nine turns later the agent recommends something for $80. The recommendation, in isolation, is a perfectly good response. It violates a constraint that lives outside the response being graded.
Every one of these is invisible to isolated grading, because the evidence needed to judge the response isn’t in the response. It’s in the rest of the session.
That’s the core of session-native evaluation: the unit of judgment is the whole session, because the unit of failure is the whole session.
From scattered telemetry to a coherent session
Here’s the practical problem. Agents don’t emit “sessions.” They emit telemetry — spans, traces, log lines — scattered across your stack, often out of order, often from multiple services, sub-agents, and tool calls that don’t obviously belong together.
So the first thing Turn3 does is reconstruction: stitching that raw telemetry back into the coherent, ordered session that actually happened. This is unglamorous and quietly hard — it’s the part that has to be right before any judging can mean anything. It involves:
- Correlating spans into a single session across services and sub-agents, even when they arrive out of order or interleaved with other sessions.
- Ordering the turns into the true sequence of what happened, including nested tool calls and sub-agent excursions.
- Extracting the through-line — the goal, the entities and facts established along the way, the constraints — so there’s a structured picture of the session’s intent, not just its transcript.
Crucially, Turn3 does this from standard telemetry you already emit — OpenTelemetry, OpenInference, OpenLLMetry — rather than a proprietary SDK. Your agents don’t change; Turn3 reconstructs the session from the traces they already produce.
The output of this stage is the thing everything else depends on: a complete, ordered, goal-aware session object. Reconstruction is the foundation. Judgment is what you build on it.
Judgment: scoring the outcome and finding the breaking turn
Once you’re working from the whole session, you can finally ask the right question — did this accomplish the goal? — and, when the answer is no, the far more valuable follow-up: where did it actually go wrong?
This is where session context changes everything. An evaluator that can see the whole session can reason the way a human reviewer does:
- It can assess goal completion against what the user was actually trying to do, not against a generic “was this a good response” rubric.
- It can trace a failure back to its origin turn — where the agent first went off the rails — rather than flagging the turn where the symptom became visible. This is the “turn 3, not turn 8” problem from the first post, and it’s only solvable because the evaluation sees turn 3 and turn 8 in the same frame.
- It can produce a verdict a human can act on — the outcome, the breaking turn, and the reasoning behind it — instead of a lone number.
The difference between “response #8 scored low” and “the agent misread the budget constraint at turn 3, and everything after inherited that mistake” is the difference between an alert and an answer. Session context is what lets Turn3 give you the second one.
Why this is a starting decision, not a later feature
A fair question: if session-native is better, why doesn’t everything work this way?
Because it isn’t something you sprinkle on at the end. If a system was built around scoring responses one at a time, treating the whole session as the unit of evaluation isn’t a setting you toggle — it changes what the system is organized around. That’s a decision you make at the foundation, not a checkbox you add later.
We made that decision at the start. It’s why “session-native” is the first word we reach for to describe Turn3 — not marketing language, but the actual shape of the system.
What this unlocks next
Evaluating the whole session isn’t just better diagnosis — it’s the substrate the rest of Turn3 stands on. Because we work from the full session:
- failures can be clustered into issues by their real root cause, not their surface symptom;
- real failures can be turned into evals that gate CI, so fixes stay fixed;
- and the same session understanding that powers judging can be applied inline, in the request path, to catch a bad response before it ever reaches a user — which is Turnguard, and the subject of a post coming up in this series.
That last one is the payoff we’re most excited about: the session understanding you built to diagnose yesterday’s failures turns out to be exactly what you need to prevent today’s. More on that soon.
See it on your own sessions
The fastest way to understand session-native evaluation is to watch it work on your own agents. Point your existing OpenTelemetry exporter at Turn3 and, in about twenty minutes, you’ll see your real sessions rebuilt turn by turn — with the breaking turn called out.
Start free at https://klimber.io — or reach us at turn3@klimber.io
Turn3 is a product of Klimber Technologies — the system of record for agent quality. Read the launch post for why we built it.