Your AI agent didn’t crash. It just sent a patient down the wrong path.
She asked one question: why was my medication denied, and how do I appeal? The agent replied warmly, fluently, and with complete certainty. It was also wrong about the reason. She’ll spend her evening assembling paperwork for the wrong appeal.
By the metrics most teams track, that conversation was a success.
You’re grading the wrong exam
“Did the API work?” is the question we inherited from a world of deterministic software. If the call returned, the job was done.
Agents broke that deal. An agent can return a perfectly formed answer and still fail the person on the other end. Infrastructure metrics tell you the system responded. They say nothing about whether the customer got what they came for.
For AI agents, uptime is a vanity metric.
How the patient got failed
Early in that session, the agent misread a payer code. PA-17 actually means step-therapy documentation is missing. The agent treated it as a medical-necessity denial.
Nothing flagged it, because the reply looked fine. Every later reply looked fine too, since each one was built on the same wrong premise. By the time the agent drafted its final answer, it was explaining the wrong denial and exposing patient details it had no business including.
This is the pattern to watch for. The root cause was at turn 3, and the damage didn’t show until much later. If you only inspect the response that looks wrong, you’ll keep missing the one that caused it.
The failures a per-response check can’t see
Some of the worst agent failures are invisible in any single message:
- Contradiction: the agent says something that conflicts with what the user told it earlier.
- Goal drift: a dozen helpful replies later, it’s solving a different problem than the one it was given.
- Dropped constraints: “nothing over $50” quietly stops mattering.
The evidence for each of these lives in the rest of the conversation. So the thing to judge is the session, not the response.
“Did the user succeed?” is answerable
That’s the idea behind Turn3, the system of record for AI quality:
- See the session. Turn3 rebuilds scattered traces into one ordered, goal-aware session from the OpenTelemetry data you already emit, with no proprietary SDK.
- Judge the outcome. Frontier-model judges score whether the user’s goal was met and name the turn where things actually broke.
- Cluster the failures. Failures group by root cause into tracked issues that are active, resolved, or regressed.
- Stop the regression. Every real failure becomes an eval that gates your CI, so a fixed bug can’t ship twice, even after a model provider quietly updates.
You stop getting a low score on a single response and start getting answers you can act on.
Then stop it in real time
Turnguard, Turn3’s session-aware guardrail layer for enterprises, applies the same session understanding inline. It evaluates responses in the request path in under 200ms (p99). Because it reads the live conversation, it catches what ordinary guardrails miss: an answer that contradicts something the user established earlier, or a response about to leak PII. In the patient example, it’s what blocked the leaking draft and delivered a safe, correct rewrite.
It already happened. You just haven’t seen it yet.
omewhere in your production traffic, an agent is misreading context right now. Turn3’s free tier lets you find it before you pay anything. Point your OpenTelemetry exporter at Turn3, and in about twenty minutes you’ll see your real sessions rebuilt turn by turn, with the breaking turn called out.
Start free at klimber.io or write to turn3@klimber.io. If your agents touch customers and money, ask about Turnguard.
You may also read: