Home / Blog / Engineering

Part 3 · Why Most AI Agents Die in Production: Durable Execution

Larry Galan
Larry Galan, Co-Founder
June 15, 2026 · 8 min read
Engineering
Part 3 · Why Most AI Agents Die in Production: Durable Execution

Most AI agents die in production for one structural reason: everything they know lives in a single running process's memory. Part 1 gave us the message array. Part 2 gave us a team of agents. All of it — every message, every subagent, the record of who is doing what — is in RAM, and a routine deploy restarts the box and erases it mid-run. The fix is one idea: write the array down. Move the source of truth out of the process and into a durable store, and the process becomes disposable. That is durable execution.

This is Part 3, in plain English and faithful to the deck below: why agents die, what durable execution actually is, how an agent comes back from the dead instead of starting over, and what that buys you. Here is the whole thing as a scrollable deck, and the written breakdown follows.

Why Most Agents Die in Production: The full deck (Part 3 of 3)Download PDF
Why Most Agents Die in Production: The full deck (Part 3 of 3), page 1 Why Most Agents Die in Production: The full deck (Part 3 of 3), page 2 Why Most Agents Die in Production: The full deck (Part 3 of 3), page 3 Why Most Agents Die in Production: The full deck (Part 3 of 3), page 4 Why Most Agents Die in Production: The full deck (Part 3 of 3), page 5 Why Most Agents Die in Production: The full deck (Part 3 of 3), page 6 Why Most Agents Die in Production: The full deck (Part 3 of 3), page 7 Why Most Agents Die in Production: The full deck (Part 3 of 3), page 8 Why Most Agents Die in Production: The full deck (Part 3 of 3), page 9 Why Most Agents Die in Production: The full deck (Part 3 of 3), page 10 Why Most Agents Die in Production: The full deck (Part 3 of 3), page 11 Why Most Agents Die in Production: The full deck (Part 3 of 3), page 12 Why Most Agents Die in Production: The full deck (Part 3 of 3), page 13 Why Most Agents Die in Production: The full deck (Part 3 of 3), page 14 Why Most Agents Die in Production: The full deck (Part 3 of 3), page 15 Why Most Agents Die in Production: The full deck (Part 3 of 3), page 16 Why Most Agents Die in Production: The full deck (Part 3 of 3), page 17 Why Most Agents Die in Production: The full deck (Part 3 of 3), page 18 Why Most Agents Die in Production: The full deck (Part 3 of 3), page 19 Why Most Agents Die in Production: The full deck (Part 3 of 3), page 20 Why Most Agents Die in Production: The full deck (Part 3 of 3), page 21 Why Most Agents Die in Production: The full deck (Part 3 of 3), page 22 Why Most Agents Die in Production: The full deck (Part 3 of 3), page 23

A routine deploy kills every agent

Crashes, timeouts, out-of-memory kills, a reclaimed container: those happen. But the failure that gets everyone is the most ordinary one on the calendar. A normal Tuesday deploy restarts the box, and every in-flight agent silently dies mid-run. This is not the rare failure. It is Tuesday.

What makes it fatal is that there is no resume. When the process dies, the array dies with it, and there is no checkpoint to fall back to, so the next run starts cold from the very first message. Picture forty minutes of work, fifteen iterations, fifty-nine tool calls, and the process dies on call fifty-eight. Fifty-eight steps of work, one death, nothing saved. The agent does not come back at 98%. It comes back at zero.

Why you can't test your way out of it

The demo gap is the prod gap. A thirty-second loop on your laptop never meets a deploy, because no deploy lands in a thirty-second window. A six-hour production job meets three of them. Nothing about the agent changed between those two runs; only the length of its exposure did.

So this is architectural, not bad luck, and no amount of testing the happy path removes it. The gap is not reliability in the ordinary sense of catching more errors. The gap is a runtime that outlives a process.

What durable execution actually is

Durable execution is a computation that survives the death of its process. That is the whole idea, and it rests on four ingredients. The useful way to group them is not by when you write them but by when they act: two you decide up front, and two that kick in at the moment of recovery.

The two you decide up front are externalized state — the truth leaves the process for a durable store — and idempotent retries, so a step that runs twice does no extra damage. The two that act at runtime are checkpointing, which records each completed step as it finishes, and automatic recovery, where a fresh process picks the work back up on its own. Two you design; two you watch work.

Decide where the truth lives

The first is the load-bearing one. Part 1's lesson was that an agent's memory lives in your code, but your code lives in a mortal process, so memory has to live deeper than code: in a store the process cannot take with it. Move the source of truth out of process memory and into a durable store, and the compute that reads and writes it becomes disposable. You can kill it, restart it, or run ten of them. Externalize the state and the process stops being precious.

A step run twice can't double-charge

The second follows from how recovery works. Recovery works by re-running, so any step that might run a second time has to be safe to run a second time. "Charge the card" twice hurts. "Set status to paid" twice changes nothing. This is the same idempotency habit from Part 2, where it kept parallel subagents from corrupting each other; here it is what makes coming back from the dead safe at all.

Resume, don't restart

The two runtime ingredients are easier shown than defined, and the distinction they turn on is the one most teams get wrong. A naive restart goes back to step one and re-fires every side effect along the way, which is exactly the dangerous behaviour. Real recovery reads the last checkpoint and continues from there, so only the work that was actually left runs. Recovery you can trust starts from the last checkpoint, not from zero.

The mechanism is rehydration. On recovery, the fresh process reads the durable record, reconstructs the message array, and continues at the last completed step. It is the same array from Part 1 — it just comes back from the database instead of from the dead process's RAM. One precision worth keeping: resume is at the step grain, not per individual tool call.

The hard case: the tool ran, the result was never written

There is one genuinely nasty case. The process died after a side-effecting call but before its result was saved. The tool ran; the row is empty. You cannot tell from the record whether it happened.

The wrong move is to silently re-run it, because that is how you charge a card twice. The right move is to flag it: mark the step interrupted, verify whether it actually landed, and only then decide whether to retry. When you can't tell if a step ran, idempotency is the only safe answer.

What durability unlocks: autonomy is just survival

Here is the payoff, and it is bigger than it sounds. Before durability, "an agent" is a demo that runs until something kills it. Autonomous means it survives long enough to finish real work. Externalizing state buys exactly one big thing — longevity — and everything people mean by autonomy follows from it. The thing that survives a deploy is the thing that can work unattended.

Take a real task: read a thousand student records and file a compliance report. That is hours long, and it was impossible the moment a mid-run deploy erased an hour's work. With durability, the job runs six hours, dies at hour two, resumes, dies at hour four, resumes again, and files the report. One continuous job across three process lifetimes, and the only thing that noticed was the log. That is what turns "an agent" into an autonomous one — and it is the shape of the work behind our university ISSS case study.

Governance falls out for free

Then a bonus arrives that you did not build. The durable record that lets the agent survive is the very same record that makes it auditable: every state change, already persisted. Same bytes, two uses. The recovery log and the audit log are the same rows.

Which means you can replay any step to verify it, pause or gate or kill a run in flight, and show every change that was ever made — not as three features you shipped, but as a side effect of the one thing you did to keep the agent alive. We wrote about that harness in more depth in Building Autonomous Agents for Regulated Workflows. You built it to survive; you got governance for free.

The durable agent, end to end

Put the trilogy together and the machine has four moving parts. The array is persisted as it grows, every step written as it happens. Leases hand work to a process and let dead work be detected and reassigned. Rehydration lets a fresh process rebuild and resume. And one database holds the single source of truth. It is the same agent from Part 1. Its memory just lives somewhere a deploy can't reach.

The honest tradeoff: durability isn't free

Every checkpoint is a write, so you pay one extra write per step. And once the truth is spread across rows, two rows that should agree can drift. So don't pay for it when you don't need it: a one-shot chat doesn't need durability, a three-second tool call doesn't, and neither does anything you can cheaply redo.

Reach for it when the work is hours long, when money or records are on the line, or when the work is simply too costly to repeat. Pay for durability only when the work is too long — or too important — to lose.

Everyone builds the same four ingredients

This is not a contested area of engineering. The industry does not disagree on what durable execution needs; it disagrees only on where the truth lives. That is the entire spectrum. At one end a dedicated engine owns the truth in a separate system, which is Temporal or AWS Step Functions. At the other end the truth lives in tables you already run, which is plain PostgreSQL or DBOS. That second end is where we sit.

None of this is new, either. It is the same lineage as workflow engines, the saga pattern, and event sourcing — and, further out, the same idea as FedEx tracking a package across handoffs. You may not need a dedicated engine. You always need state that survives the process dying.

That's the trilogy

One agent, a team, and a machine that survives. Part 1 was the anatomy of a single agent down to the API call. Part 2 was subagents as tools, an orchestrator, and one owner per shared resource. Part 3 is the state that outlives the process. If you can draw the durable agent from memory — array externalized, leased, rehydrated, with the database as the one truth — you understand this better than most of the feed.

Where Clausey fits

This is how Clausey actually runs. Our agents work through document libraries for hours at a stretch, and they do it across deploys because the truth is not in the worker — it is in the database, written step by step as the work happens. That is also why every answer can be traced: the rows that bring an agent back from the dead are the rows that show you what it did.

So the next time a vendor promises an agent that "runs autonomously across your systems," you know the only question that matters: where does the state live, and what happens to it on Tuesday?

See an agent that survives the deploy.

See Clausey read your contracts.

Bring a handful of your documents. Watch it answer, cite, and flag what your CLM never could.