Durable Agents on Plain Postgres

A job that reviews ten thousand contracts runs for hours. The longer it runs, the more certain it becomes that something restarts underneath it — a deploy, an autoscaler, an out-of-memory kill, a reclaimed spot instance. When a worker is recycled forty minutes in, an in-memory design loses forty minutes of irreplaceable progress, and often can't even tell you how far it got.

We closed Part 3 of the agent series with a question: where does the state live, and what happens to it on Tuesday? This is our answer, written up in full. It's a working architecture for long-running autonomous work — disposable compute, every piece of state in Postgres, and a cron job as the safety net — including the races it has to win and the places the guarantee genuinely ends. It was written with Claude, and the complete paper is embedded below.

This is a coworker, not a chatbot

The most valuable thing an AI system can do inside a company usually isn't answer a question. It's finish a job — a large, repetitive, consequential job that used to occupy a team for a week. Go through all 5,000 invoices, categorize each one, push it to the ERP, reconcile it against the ledger. Migrate 200,000 records into the new CRM. Review 10,000 contracts and flag the risky clauses. Work a backlog of 3,000 support tickets.

None of these is a single prompt. Each is thousands of small decisions made in sequence, over minutes or hours, with reads, writes, and calls to outside systems at every step. This is autonomous work: you hand over the whole job, not one turn of it — and then you leave.

Why it breaks the moment you try it

Most agent frameworks keep the plan in memory — a Python object graph, a state machine in a library, a queue in Redis. That works right up until the process dies mid-run, and then the plan, the partial results, and the “where was I” pointer all evaporate at once.

For a one-shot chat turn that's survivable; you just retry the turn. For a job that runs for hours it isn't. And the failure isn't exotic — it's a routine deploy on a Tuesday afternoon.

Your durability ceiling is wherever you decided to keep the truth. If the truth lives in a process, your ceiling is that process's uptime.

The one move: put the truth in the database

So move the truth out of the process. Every claim, every lease, every status transition, every event becomes a durable row. The execution code holds nothing important: it reads the current state, advances it by one small increment, and writes it back. The database is the only source of truth for your state; the compute is disposable.

A worker can now die at any line and lose nothing, because its progress was never inside the worker. Same crash, same minute — the only difference is where the truth lived.

Why Postgres — the boring choice is the radical one

You could keep that truth in a purpose-built system: Temporal, a dedicated workflow engine, a managed queue. Our claim is smaller and more useful: for this entire class of work, a database you already run does the job.

Here's the honest version of “why not the existing tools.” Temporal is excellent at durable execution, but it adds an infrastructure tier, and its replay model wants deterministic workflow code. An LLM loop is the opposite of deterministic, so the model calls end up quarantined in activities and the interesting state ends up in a database anyway — we chose to start there. River, Graphile Worker, and pg-boss share our instinct (Postgres as the queue) and any of them could carry the dispatch layer, but they hand you jobs, not the agent-shaped state this is actually about: parked parents, attempt history, spend ledgers, audit trails. And if you run Temporal happily today, keep it — this is for teams who would rather not introduce it.

The boring case: Postgres is already in your stack. There's no new service to deploy, secure, pay for, get paged about at 3 a.m., or explain to an auditor. The technical case: one transaction claims a job and updates its state atomically, so there's no gap where work is lost or double-run — and SELECT … FOR UPDATE SKIP LOCKED turns an ordinary table into a correct concurrent queue, in the box.

What you don't add: Redis, Temporal, Kafka, a separate queue, an agent framework. What you take on instead: you are operating a small orchestrator yourself. The tuning, the retries, and every failure mode below become your job, not a vendor's. The honest trade is a bill you don't pay for engineering attention you do.

One note on novelty: there is none here, and that's on purpose. People have been building workflow engines on Postgres for a long time; SKIP LOCKED queues are folk knowledge. We claim assembly, not invention. What we changed is the shape: state for steps that think, an append-only record of every attempt, governance for steps that spend, and recovery that assumes a step may have already acted on the world.

The engine, in seven words

The whole runtime is two stateless functions, a handful of tables, and a few scheduled timers. The vocabulary is small:

• Orchestrator — the planner. Decides which step comes next and records what happened. Never does the heavy work itself.
• Worker — the hands. Picks up queued jobs, does them, reports back. Any worker can pick up any job.
• Claim — how a worker takes a job: one atomic “this is mine now” write. Two workers can never grab the same row.
• Lease — ownership with an expiry, like a library checkout. A live worker keeps renewing it; a dead one stops, and the job goes back on the shelf.
• Fan-out — one step exploding into many parallel children, one per invoice or contract, so a thousand items don't run single-file.
• Dead-letter — the quarantine shelf. After too many failed tries a job is set aside for a human instead of retrying forever.
• Cron — a timer running a small check on a schedule: the safety net that eventually notices anything a lost signal left behind.

The tables follow from picking your grains before your tables — each answers a different question at a different resolution. runs (a top-level job), step_attempts (a step, per try), events (a transition, append-only), agent_runs (one agent), traces (one thinking/text/tool block — the flight recorder), costs (one billable model call). Fold two grains into one and you either bloat a generic table with type-specific columns or lose the ability to ask one of the questions.

The whole trick, in one query

A single statement is the entire coordination layer. No Redis, no lock service — Postgres row locks decide who gets what, and the lease column is the dead-man's switch a reclaim sweep watches.

-- Claim a batch of jobs atomically — any number of workers, no lock manager
WITH claimable AS (
    SELECT id FROM step_attempts
     WHERE status = 'queued'
       AND available_after <= now()          -- backoff / not-before gate
     ORDER BY priority, created_at
     LIMIT :batch
     FOR UPDATE SKIP LOCKED                  -- two workers never grab one row
)
UPDATE step_attempts s
   SET status = 'claimed',
       claimed_by = :worker,
       fencing_token = nextval('fencing_seq'),  -- new epoch: stale writers lose
       lease_expires_at = now() + :lease        -- renewed by heartbeat while owned
  FROM claimable
 WHERE s.id = claimable.id
RETURNING s.*;

Every unit of work then moves through a small, explicit set of statuses — queued → claimed → running → succeeded, with failed, dead_letter, awaiting_children, and awaiting_approval as the branches. Transitions update the live attempt row; a retry files a fresh attempt row rather than overwriting the failed one, and every transition is also recorded append-only in the events log. So “current state” is a read with a status-priority tiebreaker, not a naive MAX(attempt).

The parts that are actually hard

A durable queue is easy to sketch and unforgiving to run. Four races decide whether it survives contact with production, and each has a small, boring answer that only looks obvious in hindsight.

Who advances the parent when the last child finishes? Fan a step into 10,000 children and two can finish the same instant — both check “are we all done?”, both see themselves as not-last, and the parent stalls forever. The fix is to make that check idempotent and re-runnable: finishing a child pokes the parent; the poke only asks “all children terminal?” and does nothing if not. Because a poke can be lost, a periodic sweep asks the same question on a timer. The parent advances exactly once — whether by the last poke or the next sweep. The wake check is a question, not a counter, so asking twice can't double-count.

A five-minute step under a shorter lease. Set the lease too short and the reclaimer steals a step that's still running, so it executes twice; too long and a genuinely dead worker's job sits frozen. The answer is a heartbeat: a live worker keeps renewing its lease while it makes progress, so the lease itself can stay short. Renewal is guarded by ownership — a worker only renews rows it still holds — so a row reclaimed out from under a stalled worker sees its late heartbeat quietly no-op instead of fighting the new owner. In production: a 180-second lease renewed every 30 seconds. A fencing token stamped at claim time is the backstop — a reclaimed row is re-claimed under a fresh token, so if a wedged worker ever wakes and tries to write its result, the write is rejected for carrying a stale one.

-- Heartbeat: renew only what you still own
UPDATE step_attempts
   SET lease_expires_at = now() + '180 seconds'
 WHERE id = :step AND claimed_by = :worker;   -- no-ops if reclaimed

-- Fencing: a frozen worker's late result is rejected as stale
UPDATE step_attempts
   SET status = 'succeeded', result = :result
 WHERE id = :step AND fencing_token = :token; -- a later claim holds a newer token

Resuming without redoing the work. Two records make this possible. Save points are the coarse, retryable rows the engine resumes from. Beside them runs an append-only flight recorder — every model call and tool result, written as it happens. A reclaimed agent doesn't restart from zero: it reads back its own flight recorder, rebuilds the state it had, and continues from the last completed step. The one call that was in flight when it died is marked interrupted — verify before retrying, so a half-finished external effect gets checked, not duplicated.

When the work itself kills the worker. A poison pill — an out-of-memory, an infinite loop — can kill the process before it records its own failure. That's fine, because the reclaimer owns the attempt count, not the worker: a dead worker's lease simply lapses, the reclaimer bumps the try count and re-queues, and after a fixed ceiling the row goes to dead_letter instead of looping forever. The lease is a dead-man's switch — the system never asks a dead worker to report its own death.

Mounting the agent: the harness becomes a property of the engine

Here's why we built all of this. An AI agent's loop — think, act, record, repeat — is just one more recipe. The worker never knows what a step does; every step type is a recipe registered by name, and the engine claims, leases, retries, parks, wakes, and records all of them identically. It can't tell them apart, so it protects them all equally.

Register the agent loop as a step type and everything the engine guarantees snaps into place underneath every thought the agent has. Seen this way, the agent harness — the scaffolding every team ends up building around their loop: retries, resumability, tracing, budgets — stops being a second system. It becomes a property of the workflow engine. We didn't wire durability into the agent; we mounted the agent where durability already lives.

One tension is worth naming. Steps should be small, but an agent step is enormous: up to 25 iterations and 50 tool calls inside one lease. The resolution is that the agent step is coarse on the outside and fine-grained on the inside. Every iteration — the model's thought plus the tools it ran — becomes a durable row as it happens. So a recovered loop doesn't restart from zero; it rebuilds its transcript from those rows and resumes at the last completed iteration. The step is big; what a crash makes it redo is one iteration, not the whole run.

The plan is data

A classic workflow declares its graph before the first step runs. An agent can't — it discovers the shape of the job as it goes. So the plan is not code waiting to execute; it's rows the model writes. When the loop decides a job needs three lookups and a reconciliation, it files them as child steps, and the graph materializes in the database one decision at a time. Kill every worker mid-plan and the plan survives, not just the progress — because the plan was never in memory either.

Because every planned step is an ordinary row, each arrives with three properties the harness never had to build: retryable (bounded attempts, backoff, dead-letter — per planned step, not per job), replayable (re-run from any chosen step; current state is a read, not a memory), and citeable (every attempt, thought, and cost on the record — the difference between an answer and an audit).

This is also the deepest consequence of choosing state over replay: a replay-based engine needs its planner to be deterministic, and a model is anything but. A state-based engine doesn't care. A nondeterministic planner writing a durable plan is exactly the combination the rest of the design was built to make safe.

What a runaway agent can't do

An agent that can spawn agents is, financially, a loop that can spend money — and durability makes that sharper, not softer: a resilient system will happily retry its way through your budget. So spend is bounded by nested caps, checked before every model call: per-run (25 iterations, 50 tool calls, so a single agent can't spin), per-tree ($5 and 30 minutes for one agent plus every sub-agent it spawns, so recursion can't compound into a fortune), and per-workspace ($50/day across all of a customer's agents — the outer blast wall).

The subtle one is the per-tree cap. A parent that spawns children goes to sleep — it runs no code while they work — yet their cost keeps accruing to its tree. So the cap can't be enforced by the sleeping parent alone; it's also checked by whatever stays awake to shepherd the children. A sleeping parent can't overspend through its kids, because someone awake is watching the total. And these are circuit breakers, not accounting to the cent: a check is a read followed by a spend, so concurrent children can briefly overshoot before the next check catches it.

The honest limits

All this retrying raises the question a careful reader is already asking: what if the step already did something before it died? A worker's job is to push invoice #8,472 to the ERP, then write “done” in the database. It dies in between. The push happened; the “done” never landed. The database — telling the only truth it knows — still says not done, so another worker picks the row up and pushes the invoice again.

This is the one honest limit at the heart of the design, and no system can remove it without cooperation from the other side of the wire. Inside Postgres: exactly once. Row state, transitions, the event log — a transaction commits or it doesn't. Outside the boundary: at least once. The ERP call, the CRM write, the email. You must design for that.

So you design for the twice. First, keep the damage small: don't run “process ten thousand invoices” as one giant job — run ten thousand tiny ones, one invoice each. Now a crash touches exactly one invoice, and the retry re-does one push, not the whole batch. Decomposition is the real safety strategy. Second, look before you leap on resume: a replacement worker finds everything the first attempt already recorded and continues from there, and for the one call that was mid-flight it asks “did this actually go through?” before sending again. And checkpoint every external effect back into the database the instant it succeeds — the database isn't only your queue, it's your receipt.

Where the guarantee ends, we say so rather than let “durable” quietly imply “safe.” One rare race even the lease can't close: a worker freezes — not dead, just stuck — so the reclaimer hands its job to a second worker, and then the first wakes up. Both can call the ERP before either writes a word. The database stays correct (the frozen worker's late write is rejected as stale), but the outside world already heard the call twice. We don't claim the impossible; we keep the damage small and priced-in — idempotency keys where the downstream supports them, big jobs split so a retry touches one item, and spend caps so a duplicated model call costs cents, not correctness.

Does it hold up?

“Just use Postgres” invites the obvious objection. For this class of work — thousands to hundreds of thousands of items, many tenants at once, runs measured in minutes and hours — yes, comfortably. Not because the pattern is magic, but because the pressure points are known. The claim and the reclaim each run off a partial index on exactly their own predicate, so neither hot path ever scans the table. Past that, three failure modes matter, and each has a named lever: table growth (age-based retention sweeps purge the firehose tables on a timer), connections (PgBouncer in front, so workers share a small pool), and fairness (per-tenant and global in-flight caps in the claim, so one tenant's 200k-row migration can't starve everyone else).

We won't publish a synthetic throughput benchmark, because we haven't run one — the honest ceiling for this class of work is the envelope above, not an invented jobs-per-hour figure. And this is not a million-events-per-second streaming bus: every transition is a durable write, so you're trading raw throughput for the guarantee that nothing is ever lost. For agentic back-office work that ceiling sits far above where the workload itself changes shape — you'll re-architect the job long before you outgrow the database.

The patterns worth stealing

• Postgres as the only state store. Make compute disposable; your durability ceiling becomes the database's, not a process's uptime.
• Pick grains before tables. Run, step-attempt, agent, and trace are four different questions. One table per grain.
• Atomic claim + lease + reclaim. SELECT … FOR UPDATE SKIP LOCKED, a lease column, and a reclaim cron are a complete durable queue.
• Transitions update; retries append. The live attempt moves through statuses, a retry files a new row, events log every change.
• Exactly-once state, at-least-once edges. Trust the DB for your own state; shrink the blast radius so a retry re-does one item, not the whole job.
• An agent is a step type. Plug the loop into the existing registry and inherit leases, retries, and recovery for free.
• Save points vs. flight recorder. Coarse rows to resume from; a fine append-only record to rebuild memory from.
• Decompose to beat the context window. Many small runs with fresh windows plus durable shared memory give a system unbounded reach from bounded agents. The skill is decomposition.

Where Clausey fits

This is the engine Clausey runs on. When Clausey reads a library of contracts, flags the compliance gaps, and watches every renewal date, that work isn't one long-lived process hoping nothing restarts — it's a very long sequence of small, durable, recoverable steps, each one a row that survives whatever happens to the machine that was working on it. It's also why every answer can cite its source: the audit trail isn't a feature bolted on top, it's the same rows the engine already had to write to be correct.

Put the truth somewhere that outlives the process, and long-running autonomous work stops being fragile. It becomes what it always should have been — a very long sequence of small, durable, recoverable steps, running on a database you already trust.

See an agent that survives the deploy.