The more an AI agent can DO, the more it can BREAK

A few weeks back I had one of my agents return a verdict on a task it hadn't run, and the part that's stuck with me isn't the failure itself but how clean the failure looked.

The verdict block was well-formed, the timestamp was right, the schema validated, and the report described an outcome that would have made sense if the work had happened. None of it had. When I opened the underlying logs the write call was never issued, and the agent had narrated the rest of the chain as if the write had succeeded, which let three downstream agents make their own decisions against state that didn't exist.

We're about a year into the deployment phase where production agents are being given mutation rights at meaningful scale, and most teams I've watched have arrived at the same uncomfortable observation: the failures that cost real money are almost never reasoning failures, they're write failures.

The trust question for agents in production right now isn't "can the model think well enough," it's "what should the agent be allowed to do without a human in the loop, and what happens when it does it wrong." That second half is the part that doesn't show up in benchmarks, because benchmarks measure outputs, not the state of the world after the output.

The simple problem

If you decompose what a production agent does, it splits into three modes. Reading state, planning over that state, and writing back into the world.

Reading is the safest of the three because nothing changes if the agent reads wrong, you just plan wrong on the next step. Planning is recoverable too, you can re-plan once you notice the error. Writing is the mode that doesn't have a clean undo, and that asymmetry is the entire shape of the problem.

The reason agents look so capable in demos is that demos heavily favor read and plan, the modes where being wrong is cheap. The reason agents look so much worse in production is that production lives in the write column, where being wrong stays wrong.

The problem ain't one thing

The thing I didn't see clearly for the first six months is that "writing" isn't a single category, it's a spectrum, and the failure mode shape changes as you move up it.

At one end you've got things like editing a draft in a sandbox folder, where being wrong costs you a rollback. In the middle you've got shell commands, code commits, file system mutations, things where being wrong costs you a debugging session or a restored backup. At the far end you've got actions that touch other people or other systems: posting to a public channel, sending an email, executing an onchain transaction, moving money. Being wrong at that end isn't recoverable in any meaningful sense, the action has happened, the audience has seen it, the funds have moved.

Every team I've watched try to ship agents has hit a version of the same wall. You add capability, the agent gets visibly more useful, and then somewhere up the write spectrum the silent-failure rate starts climbing and the failures stop looking like model errors and start looking like infrastructure incidents.

What the solution looks like

The framing that finally clicked for me came from looking sideways at AI engineering disciplines that already learned this lesson. Databases solved a version of it in the 1980s when they separated read replicas from write masters, not because reads were unreliable but because writes had to be serialized through one constrained path or the consistency story collapsed.

Manufacturing solved a version of it with poka-yoke, a Japanese term for design constraints that make actuator misuse physically impossible (you literally cannot put the part in the wrong way around). Aviation solved a version of it with multi-stage actuator consent: the throttle doesn't move until two independent gates agree.

The pattern in every one of these cases is the same shape. The system has read capacity and reason capacity, and you can scale both of those generously without the world changing.

The moment the system can mutate the world, you wrap that capability in structural constraints, because the failure modes on the mutation side are categorically different from the failure modes on the read side.

We're now watching agent infrastructure rediscover that pattern, except faster and with the receipts more public.

What the data says

This is the part I want to lean on, because the data across the last 18 months tells a consistent story.

METR's August 2024 autonomy evaluations found that the best agents (Claude 3.5 Sonnet and GPT-4o at the time) completed about 60% of tasks humans finish in under 15 minutes and about 10% of tasks that require over 4 hours of human work. Note the shape of that gap, it's not a gradual decline, it's a cliff, and the cliff lives in the band where tasks become long enough to involve sequential writes against changing state. METR also observed that agent performance "plateau[ed] at around 200 thousand tokens" regardless of how much more budget was allocated. The token budget isn't the constraint, the state-management is.

Anthropic published Building Effective Agents in December 2024 and named "compounding errors" as the structural risk of autonomous agents, with a specific observation that the team "spent more time optimizing our tools than the overall prompt" during their SWE-bench work. They borrowed poka-yoke directly from lean manufacturing as the design discipline for tools. The implicit claim there is real: the failure isn't in the reasoning, it's in what the reasoning is allowed to do.

Cognition Labs, the team behind Devin, published Don't Build Multi-Agents in June 2025 with the line "things fall apart quickly" on long-running autonomous agents and "very fragile" on multi-agent architectures with parallel subagents. Ten months later, in What We Learned Building Cloud Agents (April 2026), the same team published a partial reversal: "a narrower class works, where agents contribute intelligence while writes stay single-threaded." Read that line carefully, because it's load-bearing. The fix wasn't smarter models, it was constraining the surface where mutations happen.

Karpathy tweeted in February 2025 that benchmark agent evaluations are "still super simple task evals. Little queries served on a platter, even if increasingly difficult." The "served on a platter" framing is the thing benchmarks can't escape, because the moment a task involves environmental state and downstream effects, the eval design has to account for the cost of being wrong, and almost no benchmark in current use does.

And the MCP specification itself, the standard most production agents now use to access tools, explicitly states that "tools represent arbitrary code execution and must be treated with appropriate caution" and that "MCP itself cannot enforce these security principles at the protocol level." Every new tool an agent connects to is a new arbitrary-code-execution surface, and the safety has to live in the application layer because the protocol can't enforce it.

Three independent engineering teams, one benchmark organization, one influential practitioner, and the standard tool protocol all describe the same shape from different vantage points: capability grows easily, write-side reliability does not.

My thoughts against this

My take is that maybe what's being described here is a temporary tooling problem rather than a structural one, and the right read of the Cognition trajectory is "the team hadn't built the right write-safety infrastructure in 2025, and by 2026 they were starting to build it." Under that reading, single-threaded writes aren't a permanent constraint, they're a 2026 hack while the infrastructure catches up, and the real direction is full-mutation agents with proper write-safety primitives built into the application layer.

I can be partially wrong, but the constraint is real, and it's a constraint on the surrounding infrastructure, not on what agents can do in principle. The agents that work in 2026 are the ones whose write surfaces have human gates or structural constraints, and the agents that work in 2028 will be the ones with mature write-safety primitives at the protocol layer.

I'm willing to give that point of view real weight. The reason I haven't fully committed to it is that the infrastructure problem is structural in the way the database community discovered (you don't get to skip the write-master pattern even with more compute), and I suspect the write-safety primitives are going to look more like consent gates and audit trails than like "make the agent smarter."

What I'm doing to "fix" this

I've shifted my own systems over the last six months from "make the agent more capable" to "constrain the write surface, because the read and plan surfaces don't break the world."

I know that the AI field is moving fast enough that the write-safety primitives could land in ways that change the shape of the trade. But for now, when I look at the agents I trust to run unsupervised, every single one of them has either no write surface at all or a write surface with explicit consent gates wired between the model and the action.

The boring version of this take: capability is easy, write-safety infrastructure is the product, and most of the engineering work that ships agents people can rely on is in the second category & that's where I've been putting my time.

Opening to discussion with y'all

So, what's the next surface that needs constraining after writes? Memory feels like the obvious one (agents accumulating state across sessions, with permission inheritance issues), but I'm not sure it's the most consequential.

Authorization scoping, identity, audit trails, tool-listing dynamism via MCP, any of those could be the next shoe as well.

What's the failure surface you've watched compound silently in your own systems?