Repair Budgets Are Durable State, Not a Counter

The moment you let an automated system retry its own failed work, you have introduced a loop. And a loop with no bound is a liability. So you add a budget: three attempts, then stop and escalate. That part is easy to describe and easy to get wrong, because the hard question is not how many retries you allow — it is where the count lives. If the answer is "a variable in the running process," you don't have a bounded loop. You have a loop that resets every time the process dies.

This is a note about the parts of an automatic repair loop that only show up when you take restarts, ambiguity, and process boundaries seriously.

The problem

Assume a controller that runs work, checks it, and — if the check fails — sends the work back to be redone, with feedback, up to some fixed number of times. In prose this is a five-line description. In practice it is several coupled decisions that all have to agree with each other:

• Where does the attempt count physically live?
• How does the reader of that count and the writer of that count agree on what they're reading?
• What happens if the controller crashes between "this attempt failed" and "record that it failed"?
• What happens to the evidence of the failure while the work is being redone?
• How do you prove the feedback actually reached the thing doing the repair?

Get any one of these wrong and the system will still pass its ordinary tests. It will just also, occasionally, loop forever, or forget why it was looping, or hand a repair worker no idea what to fix.

What actually happened

A repair budget that started life as an in-memory counter had to be reconsidered from the ground up once restart behaviour was taken seriously. Several problems surfaced that a "just increment a number" design never has to confront.

The count has to be durable, and it has to have one home. Persisting the attempt count is obvious once you say it out loud. Less obvious: the count needs one canonical structural location, not a value that the system finds by matching a field name wherever it appears. If a reader will accept an attempt count from anywhere in a stored document, then a stray, valid-looking value somewhere else in that document can be picked up as authoritative. That is not a cosmetic parsing quirk — it is untrusted or ambiguous data being allowed to set a safety limit. The budget is precisely the thing an attacker or an accident would want to reset.

Reader and writer must share one schema resolver. If the code that writes the count and the code that reads it each carry their own idea of where the count lives and what shape it takes, they will eventually disagree. The durable fix was to centralize both behind a single locator, so that "where is the trusted value" has exactly one answer that reader and writer physically share.

The accepted transition must be one atomic write. When work is finally accepted, two things happen: the attempt counter resets, and the item advances to the next stage. If those are two separate writes, there is a crash point between them — and a restart into that gap leaves you with an item that has advanced but kept a stale count, or reset its count but not advanced. Both are wrong in ways that are hard to detect later. The transition has to reset the counter and advance the stage in a single durable write, so a crash lands you cleanly on one side or the other.

You must distinguish absent from corrupt. "No count yet" and "a count I can't parse" are different situations that demand different responses. Collapsing them — treating unreadable state as if it were a fresh start — quietly hands you an unbounded loop.

Failure evidence must be copied, not moved. When a check fails, its output is evidence, and that evidence is also an input to the next repair attempt. If you archive it by moving it, you have removed the live artifact the repair worker still needs to read. Archival here has to be copy-not-move: the feedback stays in place until the repaired worker has actually consumed it, and a verified copy goes to the archive.

The feedback has to be proven at the real boundary. The most instructive miss: an early version claimed it had proven the repair feedback reached the worker, but the test only checked that the feedback was present in the assembled context one layer up. "The context contains the feedback" and "the subprocess received the feedback" are different claims. The proof only became real when a test captured the actual hand-off to the subprocess and confirmed the data crossed that boundary.

The lesson

A bounded repair loop is not a counter with a limit. It is a small durable-state system, and every property you care about — that it stops, that it remembers why, that it survives a restart, that it feeds the repairer — is a property of the state behind it, not of the control flow in front of it.

Two failures in this story landed differently but rhymed. One was an overstated proof: a test that validated a layer above the boundary that mattered. The other was an over-broad read: a parser that accepted the trusted value from too wide a scope. Both are the same underlying mistake — being imprecise about where something is true. Feedback is true at the subprocess boundary, not at context assembly. The budget is true at one canonical location, not anywhere the field name appears.

The broader principle

Any time an automated system can act on its own prior output — retrying, repairing, escalating — you have created durable state that governs future behaviour, whether or not you designed it that way. The safe move is to treat that state as a first-class thing:

• Give each durable field one canonical location and one shared resolver for reading and writing it.
• Design every transition as: precondition, a single durable effect, and a defined restart behaviour if you crash mid-transition.
• Distinguish absent state from corrupt state, and choose the fail-closed response to each on purpose.
• Preserve any artifact that is still an input. Copy to archive; don't move.
• When acceptance depends on data reaching an external worker, prove it at that worker's boundary, and claim no more than your test actually demonstrates.

How to apply it

Before you build a loop like this, write down the durable-state checklist and answer it explicitly:

1. Canonical location. Exactly where does the budget live, and is there exactly one place a reader will look?
2. Reader/writer symmetry. Do the reader and writer share a single resolver, so they cannot disagree about the trusted representation?
3. Atomic transition. Does acceptance reset the counter and advance the stage in one write, with a defined outcome for a crash at every point?
4. Absent vs corrupt. Are "no state" and "unreadable state" handled as distinct cases?
5. Evidence survival. Does the failure evidence survive — by verified copy — until the repaired worker has consumed it?
6. Boundary proof. Is there an end-to-end test at the real dispatch boundary, plus a negative control that fails if the intended behaviour is removed?

Then test the adversarial layouts, not just the happy path: a stray field outside its section, duplicate sections, malformed values, missing anchors, a partial archive, a restart mid-transition. A repair loop that has only ever been shown to work on clean, valid, single-process state has not been shown to be bounded at all — it has been shown to be bounded on a good day.