Recover Durable Workflows with External Effects and Long Waits

Design durable execution and recovery for a distributed DAG workflow system. Workers can crash or lose connectivity, tasks can have irreversible external effects such as billing or sending a message, and some nodes wait for a webhook or human approval for several days. The entire scheduling cluster may restart while many runs are unfinished.

Constraints & Assumptions

• A lease expiry can cause duplicate dispatch even while an old worker continues running.
• The source asks for end-to-end exactly-once external effects. Evaluate whether this is achievable for each external service contract rather than assuming the scheduler alone can guarantee it.
• Waiting for a callback or approval must not occupy a worker thread for days.
• Recovery must preserve the run's dependency graph, completed outputs, pending work, and unresolved external operations.

Clarifying Questions to Ask

• Can each external destination deduplicate operations or participate in a suitable transaction?
• Can the status of a timed-out side effect be queried by a stable operation ID?
• Are workflow definitions immutable for an existing run, and where are task outputs persisted?
• How are callbacks authenticated, correlated, deduplicated, and ordered against timeouts or cancellation?

Part 1 — External Effects and Duplicate Attempts

Describe durable intent, attempt ownership, completion, and retry behavior. Explain the difference between exactly-once logical state transitions and exactly-once effects at an external destination.

What This Part Should Cover

• Stable logical operation identity separate from worker attempt identity.
• Fenced local writes and destination-enforced deduplication where available.
• An explicit uncertain state when neither safe retry nor confirmed success is established.

Part 2 — Durable Waits

Design webhook and human-approval nodes that can wait without retaining an executing worker. Explain registration, callback races, repeated callbacks, deadlines, and resumption.

What This Part Should Cover

• Persisted correlation, authorization, deadline, and wait state.
• Atomic selection of one terminal wait outcome.
• Durable readiness of downstream tasks after resumption.

Part 3 — Full Scheduler Recovery

Explain how a replacement scheduling cluster reconstructs unfinished runs after complete loss of scheduler memory. Distinguish tasks that can be dispatched safely from attempts whose external outcomes remain uncertain.

What This Part Should Cover

• Definition versions, task states, dependency completion, and durable artifacts.
• Recovery ownership and bounded scans or replay.
• No blind reset of every running task to a fresh execution.