A Stage Name Is Not an Operation Identity

Long-running automated work usually persists its position as a stage name. That is enough to describe where the work got to, and not enough to describe what it was doing. When a single stage can stand for more than one pending operation, a crash in the wrong moment leaves you holding a label instead of a decision — and the safest-looking recovery is often the wrong one.

The problem

Automated workflows persist their position so work can be picked up after an interruption. Most of the time that position is a coarse stage marker: the name of the phase the item had reached.

This works while each stage name maps to exactly one thing that could have been happening. The moment a stage can represent several distinct pending operations, the marker stops being a recovery signal and becomes an ambiguity.

The gap is usually created by ordering that looks entirely sensible. A workflow finishes a phase, accepts the review of that phase, advances the durable marker to the next phase, and then creates and starts the work for it. Between the marker write and the work actually starting there is a window. It may be milliseconds; it may be long enough to set up a working directory and launch a subprocess. If the process dies inside that window, everything durable says is next phase.

A generic resume reads that and does the most reasonable thing available to it: it treats the item as entering that phase normally. Entering normally is precisely what must not happen. The item was not entering the phase; it was part-way through one specific operation whose inputs had already been reviewed and accepted.

The consequences all follow from the same ambiguity:

• Earlier, already-accepted phases can be re-run.
• An operation that actually completed, or actually failed, can be restarted through generic routing.
• Two concurrent resumes can each conclude the work needs starting, and both start it.
• Recovery can proceed from a different source revision, or a different review outcome, than the one that was approved.
• The component performing recovery can end up writing state that another component is supposed to own.

None of those are retry problems. Adding a retry flag to the failed operation would not have addressed any of them.

What actually happened

The fix was a durable continuation record: before the stage marker advances, the workflow writes down the operation it is about to perform. Not "we are now in phase four", but "a specific operation of this type is pending, derived from this exact accepted source, under this exact review outcome, on this attempt". Recovery then reads a description of pending work rather than inferring one from a phase name.

Getting to that design took more revision than expected, and the revisions are the interesting part.

Review of the behavioural contract caught, across several passes:

• Terminal records falling through. A continuation record marked completed or failed would drop out of continuation handling and land back in generic phase-entry routing — reintroducing the original defect for the exact cases the record existed to describe.
• A rule that could not be implemented. The definition of "this operation is still actively running" relied on a check that had no reliable implementation on the path where it actually mattered.
• A token given more authority than it could carry. The design used a process identifier as an ownership claim. A process identifier does not establish ownership; it only names a process that may or may not still exist, and may or may not be the same process.
• A boolean that concealed two different answers. "Is the previous owner alive?" returning false was being used to mean proven dead, when it could equally mean the check itself failed unexpectedly. Those two must not authorize the same action.
• Acceptance wording that outlived the design. One criterion still referred to a mechanism the design had already replaced.

Review of the implementation ordering caught something different again: the first plan made the continuation path reachable before the safety check that guarded it was installed. The end state was sound; the intermediate state was not.

And review of the finished implementation caught a real concurrency defect that had survived both of the earlier reviews. Admission — deciding a resume is allowed to take this operation — and claim — writing down that it has taken it — happened in two separate lock-held transactions. The claim then wrote from the snapshot read during admission, without re-proving that the durable record still matched. Two resumes could each be admitted against the same record and each go on to claim it and start the work. The repair was to re-read the record under the claim lock and compare it field by field before writing anything, and to move the decision about what happens next into the same lock-held transaction that records completion.

That progression is worth noting on its own: three review layers looking at three different things — the contract, the implementation order, and the finished code — each found a defect the others could not have found.

The lesson

Seven things generalize, and they compose into a single design rather than a list of fixes.

Record the operation, not just the stage. If a stage can host more than one distinct pending operation, the stage name cannot be the recovery signal. Persist the operation: its type, its status, the attempt number, and where its working state lives.

Bind recovery to the exact inputs that were approved. A continuation record should name the source revision it was derived from and the review outcome that authorized it. Recovery that resumes from whatever is current is not resuming; it is starting something new that looks similar.

Admission and claim are one logical transaction, even when setup has to happen in between. External work — creating directories, launching processes — often cannot run while holding a lock. That is fine. What is not fine is trusting a snapshot taken before the lock was released. Re-read under the claim lock and compare the whole canonical record. If anything has changed, refuse before writing.

Liveness has three answers, not two. Asking whether a previous owner is still running can return: definitely active, definitely absent, or unknown. Only positive evidence of absence should authorize taking the work back. On a POSIX system, a signal-zero probe makes this concrete — ESRCH proves no such process exists, a successful probe or EPERM both mean a process is there, and any other outcome is indeterminate and must fail closed. Collapsing those into a boolean is how you get two processes doing the same job.

Install the guard before the reader can reach it. Ordering matters as much as the final design. If a classifier can route into a path before that path's safety check exists, there is a window in which the unsafe behaviour is live. Sequence the work so the guard is installed and proven first.

Terminal states must not fall through. A record that says completed or failed has to be handled explicitly and terminally. Anything that drops out of specific handling lands in generic handling, which is exactly the behaviour you were trying to avoid.

Records from before the mechanism existed should fail closed. Older items will have no continuation record. They should refuse and ask for a human, not receive guessed continuation semantics.

The broader principle

Durable recovery is a transaction and concurrency problem, not a retry problem.

Framing it as a retry problem leads you to a flag on the failed step. Framing it as a transaction problem leads you to ask which operation was in flight, what authorized it, who currently holds it, and what it means for two callers to arrive at once. Those are the questions that determine whether recovery is safe.

There is a second, quieter principle in how the design was reached. A recovery rule can read as entirely reasonable in prose and still be unimplementable — because the seam it depends on does not exist on the path that matters, or because it needs information the caller cannot supply. Reviewing a design for internal coherence is not the same as reviewing it for implementability, and the second question catches things the first never will.

A third: concurrency claims need adversarial tests, not concurrent ones. Running two operations at once and observing that nothing broke proves very little. Constructing the specific interleaving where both callers pass admission before either one claims is what actually demonstrates the guard holds.

How to apply it

If you are adding recovery, resume, or continuation behaviour to a workflow that persists its position:

1. Model the interrupted operation before designing the mechanism. Write down what a pending operation needs to identify itself: type, status, the source it derives from, the decision that authorized it, attempt number, current owner, where its working state lives, and what happens next when it finishes.
2. Write the crash matrix first. List every gap between persisting intent, advancing the marker, creating working state, starting the process, finishing, and recording the outcome. Each gap is a state something can be found in. Each needs a defined recovery.
3. Define reclaim as a three-way decision. Active, proven absent, indeterminate. Only proven absent authorizes taking over. Indeterminate refuses conservatively and asks for a human — and you should say plainly, up front, that this is the accepted trade-off.
4. Make claiming a compare-and-set. Re-read under the lock, compare the complete record, refuse before writing if anything differs. Never write from a snapshot that predates a lock release.
5. Derive the next step inside the same transaction that records completion. If completion and routing are decided from two different reads, they can disagree.
6. Prove duplicate-start prevention with deterministic interleavings. Not timing-dependent stress tests.
7. Sequence guards before readers, and plan the rollback in reverse. Disable readers before removing the guards or writers they depend on.
8. Handle pre-existing records explicitly, at the start. Backward compatibility rules added late tend to be added as guesses.
9. Keep the scope to one operation. The pull toward a general replay or event-sourcing architecture is strong, and it is a much larger, much riskier change than the one you need. Recovery for one specific interrupted operation is a bounded piece of work; a general replay engine is a new system.

The last point deserves emphasis, because it is where this kind of work most often goes wrong. Everything above is achievable in a narrow change to one phase of one workflow. It stops being achievable the moment "make recovery safe here" becomes "build a durable execution engine".