IThe moment I stopped waiting for the transcript
For a long time, supervising my agents meant handing down a verdict. A run finished, I read the transcript end to end, and I marked it — pass or fail, ship or redo. Then I stopped grading at the end and started watching runs while they were still happening. That one change caught a category of failure the final grade never surfaced: runs that returned a clean-looking answer sitting on top of reasoning that had quietly gone sideways three steps earlier.
That's the real version of "human-in-the-loop." Most people picture a person signing off on a finished output. The version that actually earns its keep is the opposite — a person watching where a run is heading and stepping in mid-flight, before the mistake is locked in. I run a fair amount of this; if you want the plumbing underneath it, I've written up how I run multi-agent orchestration without a platform. This piece is about the supervision model that sits on top.
IIWhy grading the finished run was always going to miss things
The single pass/fail grade has a structural problem, and it's just arithmetic. A synthesis of harness-design notes from Anthropic, OpenAI and LangChain makes the point cleanly: a ten-step agent running at 99% success per step still only lands around 90.4% end-to-end, because per-step errors compound. Ninety percent sounds fine until you remember that's one run in ten quietly wrong — and the grade at the end tells you that it broke, never where.
The chains I run go well past ten steps, so the compounding is worse, not better. A verdict on the transcript is a smoke alarm that only goes off once the house is already gone. It can't point at the step where a tool call drifted, or the moment the agent started contradicting its own earlier reasoning. For the honest accounting of what running these loops in production actually costs you, here's an honest verdict on running agent loops in production.
IIIWhat "watching it work" actually means in production
Watching a run means checking, on every step, where it's heading — not scoring the transcript once it's done. LangChain published a case study on a financial-planning agent that made exactly this move: from outcome labels observed only at the end of a conversation to turn-level state inference computed on every turn, inside the latency budget of a single reply, so the system could act on where the conversation was going while it was still going there.
There's a name for the discipline underneath, and it isn't evaluation. As practitioner literature frames it, agent observability is the real-time monitoring of what the agent did, how it reasoned, and which tools and data it touched — while LLM evaluation only tests capability before or after a run. The distinction matters because agents break the assumption ordinary software runs on. Confident AI's 2026 guide puts it well: normal monitoring treats a 200 status as proof the system did the right thing, but an agent can return a flawless-looking success while its decision chain did real harm. The answer looked right. The reasoning wasn't.
IVThe three things I check while an agent is still running
My own loop borrows its shape from that case study's three-part pattern — state inference in the request path, a versioned policy per state, and offline verification before anything ships live. In plain terms, while a run is happening I'm asking three questions:
That's not theory I lifted from a slide. LangChain describes the same operational loop in that case study: state inference running in the request path on every turn, a versioned policy per state — a prompt insert, a tool choice, a template change — recorded so the team always knows which change fired, and offline verification before any change ships live. The recorded-which-change-fired part is what turns a mystery into a diff.
VSignals worth watching mid-run vs. signals only visible after the fact
Not every signal is available at every moment, and that's the crux of the whole argument. Some things you can only see while the run is moving. Others you can only see once it's stopped — by which point steering is off the table.
Doom-loop detection is the one I'd put money on first. LangChain's own coding agent is a clean proof point: harness-level changes — self-verification prompts, better context injection, and middleware that catches doom loops mid-run — took it from 52.8% to 66.5% on Terminal-Bench 2.0, Top 30 to Top 5, without touching the underlying model, using tracing to surface failure modes as they happened rather than at final scoring. Same model, better watching, meaningfully better results.
VITreating supervision as a decision, not a scorecard
Once you're watching, supervision stops being a grade and becomes a decision. LangChain's docs describe a human-in-the-loop middleware that pauses a run when a proposed tool call matches a policy you set, and hands you four moves: approve it as-is, edit it before it runs, reject it with feedback, or respond directly.
Read those options next to a pass/fail grade and the difference is obvious. A grade happens after the fact and changes nothing about the run it judged. A decision happens inside the run and changes where it goes next. This isn't an endorsement of one library — it's about where the checkpoint sits. Put it after the output and you're a critic. Put it inside the run and you're a driver. The skill this rewards is judgment applied in the moment, which is exactly the shift I wrote about in the week the reviewer became AI's most coveted role.
VIIThe oversight-theater test — where I actually spend my attention
The trap on the other side is checkpointing everything. There's a line I keep coming back to: if a human would approve something 99 times out of 100 without changing anything, the checkpoint isn't oversight — it's theater. A gate you rubber-stamp is worse than no gate, because it costs attention and buys nothing but the feeling of control.
The scale math forces the issue. Gartner expects 40% of enterprise applications to include task-specific AI agents by the end of 2026, up from under 5% in 2025. You cannot hand-review every run of that, so the only question worth answering is which specific moments deserve a human look. Most teams get there the hard way — reactively. AvePoint's 2026 State of AI report found 95.5% of organizations took at least one action after an agent-related security incident, and adding human-in-the-loop controls was the most common fix by a wide margin — oversight bolted on after an agent had already acted on a bad call. The governance gap is real too: Deloitte's 2026 State of AI in the Enterprise report sees agentic adoption climbing toward 74% of enterprises within two years, while only about one in five have a mature model for governing autonomous agents at all. Spending attention on the drifting moments, not on every tool call, is how you close that gap without drowning.
VIIIThe honest verdict: what this doesn't fix
Now the honest part, because I don't sell the fairy tale. Watching runs in real time is not free. It means tracing every step, maintaining a versioned policy per state, and — the part no tool solves for you — someone actually watching. It also doesn't retire end-of-run checks; you still want a final grade, you just stop pretending it's your only line of defense.
And it only pays off on the right kind of work. The 2026 Stanford Digital Economy Lab report found agentic implementations delivered median productivity gains of 71%, versus 40% for non-agentic tools — but tied the best results specifically to tasks with recoverable errors and clear success criteria. Real-time oversight is what makes an error recoverable; it does nothing for a task where the first wrong move is already irreversible. Regulators are converging on the same shape: the EU AI Act's Article 14 makes human oversight a legal requirement for high-risk systems, and the 2026 guidance describes a continuous standard — people who can monitor for anomalies, interpret outputs, override or reverse them, and stop the system safely — not a single sign-off at the finish. If you've lived through a bad stretch of this, you already know why continuous is the operative word; the long version is what I learned from a week of AI agent outages.
IXWhere to start if you're still grading at the end
Don't rebuild your stack to fix this. Pick one multi-step agent — the one whose failures cost you the most — and define two or three states it moves through. Just those. Watch which state it's in and whether the right response is active, and give yourself one place to step in when it drifts.
That's the whole starting move. You're not buying an observability platform; you're changing where you point your attention — from the transcript after the fact to the run as it happens. Grading tells you how the story ended. Watching lets you change it.
— Michael Rouveure
