Exhausted Automated Loops Need a Separate Recovery Authority

When an automated loop exhausts its retry budget, the safe next step is not another turn through the same controller — it is a distinct recovery path with its own admission rules. Most automated workflows have a retry loop somewhere, and the interesting question is rarely what happens inside the limit. It is what happens the moment the limit is exhausted and a human steps in to fix the underlying problem. If your answer is "re-run the same loop," you probably have a latent way to destroy the human's fix.

The problem

Picture an automated step that reviews a plan and either accepts it or asks for changes. It has a fixed rework budget — say a handful of attempts. When the budget runs out, the loop stops and waits for a person.

A person can now do exactly the right thing: read the failures, correct the plan, and ask the system to try once more. The trap is in how "try once more" is implemented. The obvious move is to re-enter the ordinary controller that runs the loop. But that controller was built to run the loop, and a fresh run of it tends to start the loop over — including resetting the counter that said "you are out of retries."

So a single new failure after the human's fix can do far more than fail. It can archive the corrected work, reset the item to the start, and dispatch the original author again — silently unwinding the human correction the recovery was supposed to protect. The surface request was "give me one more audit." The real request was "let me re-enter a safety-critical state machine without reopening the budget I just exhausted."

What actually happened

We treated the "one more attempt" feature as small and quickly discovered it was not. The visible behaviour was tiny: authorise one review, run it once, advance only if accepted, otherwise stay put. The hidden behaviour was where the engineering lived:

• durable proof that the earlier limit refusal actually happened, so recovery can't be conjured out of nothing;
• authorisation bound to the exact work, the exact corrected artifact, and the human who approved it;
• single-use authorisation, so one grant can't be replayed into many runs;
• concurrency control, so two attempts can't both "win";
• crash reconciliation, so an interrupted attempt lands in a defined state rather than an ambiguous one.

The decision that made the rest tractable was architectural: recovery got its own command and its own authority, rather than a flag bolted onto the normal controller. That boundary is the whole lesson. The recovery path was allowed to reuse the canonical review machinery — the same assembly, execution, and result-parsing primitives — but it owned admission and result-persistence itself. It never became a second general-purpose controller, and it never got the loop's permission to reset the counter.

One structural change made the reuse honest. The shared "advance to the next stage" logic assumed it could take a lock the recovery path was already holding. Rather than duplicate the stage-write code or nest the lock, we extracted a variant that runs the same transition under a lock the caller already holds. The specialised path reused the shared implementation instead of copying it — which is the difference between a recovery feature and a shadow copy of the whole workflow.

The lesson

An exhausted automated loop and its recovery are two different authorities:

• The loop decides, within a budget, whether work passes. Its job is to iterate and, eventually, to stop.
• Recovery decides whether a specific, human-corrected piece of work earns exactly one fresh, evidence-backed attempt. Its job is to admit that one attempt under strict conditions and then get out of the way.

Collapsing these into one controller is what creates the danger. The loop's instinct is to reset and retry; recovery's instinct must be to grant narrowly and never reopen the budget. When they share a code path, the loop's instinct wins by default, usually at the worst possible moment.

This connects to two ideas worth designing for explicitly. First, the budget that governs retries has to be durable, not just a counter living in process memory — otherwise every fresh invocation starts life believing the limit was never reached. Second, recovery has to demand positive proof that the current attempt actually produced fresh output, because a stale result from a previous run is often already sitting on disk waiting to be mistaken for success. Both are their own topics; here the point is narrower: even with a durable budget and fresh-artifact proof in place, you still need a distinct authority to consume them safely.

The broader principle

If a loop can exhaust itself, its recovery path is part of the design, not an afterthought you improvise during an incident. A loop without a deliberately separate recovery authority is not finished — it just hasn't failed in the specific way that reveals the gap yet.

There's a sharper version of this for anything safety-relevant. You usually cannot prove exactly-once execution across a crash at a process boundary. What you can build is a defensible contract: one durable claim per authorisation, no automatic re-dispatch after that claim, proof that fresh output was written, and deterministic reconciliation when an outcome was recorded but final handling was interrupted. That is a weaker guarantee than "exactly once," and it is the honest one. Recovery authority is where that contract lives.

There's also a humbling constraint. Recovery can only use evidence that was captured at the time of failure. In one case, an older failure could not be recovered at all, because the durable refusal record recovery depends on did not exist when it failed. The recovery command refused rather than manufacture eligibility from archived leftovers. That refusal was correct: recovering a system safely is bounded by what that system chose to record before it broke.

How to apply it

• Design the recovery path before you call the loop done. For any budgeted retry loop, write down what a human does after exhaustion, and which authority admits their fix.
• Give recovery its own entry point. Don't reach for --resume on the normal controller. A separate command can reuse the loop's mechanics while owning admission and persistence itself.
• Never let recovery reopen the budget. The exhausted counter is a fact. Recovery grants one bounded attempt on top of it; it does not reset it.
• Record refusals as durable, structured evidence, not just log lines. Anything that might need recovery later has to leave behind, at failure time, the proof recovery will require.
• Write the crash-state table before the code. Enumerate: before dispatch, dispatch claimed, work failed, output written, transition failed, completion written — and define reconciliation for each.
• Expose shared transitions as held-lock primitives. So specialised paths reuse the real state-write logic instead of duplicating it and drifting.
• Test the ugly paths explicitly. Stale pre-existing output, one winning and one losing concurrent attempt, replay of a used authorisation, and interruption mid-transition. These are the cases that quietly corrupt state.

The visible feature is "let me try once more." The real feature is a separate authority that can safely touch a state machine after its ordinary safety limit is spent. Build that authority on purpose, or discover you needed it during an outage.