Katkat
Back

Design a Refund Workflow as a Directed Acyclic Graph

DoorDashSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Design a resilient, stateful workflow orchestration engine to process complex order refunds as a Directed Acyclic Graph (DAG), ensuring idempotency, crash recovery, and financial consistency.

Scenario

At a food delivery platform like DoorDash, refunding an order is rarely a single API call. A refund may require updating multiple independent ledgers (customer payment refund vs. platform credits), clawing back merchant payouts, evaluating dasher compensation, updating fraud scores, and emitting notifications. Certain operations can run concurrently (e.g., notifying the customer and sending an alert to merchant operations), while others must run sequentially (e.g., verifying remaining refundable balance before executing a payment gateway call).

Design a durable workflow engine that executes refund requests modeled as a Directed Acyclic Graph (DAG).

Requirements

Functional Requirements

  • DAG Definition & Execution: Define workflow steps with dependencies. Independent steps must execute concurrently; dependent steps must wait for predecessors to complete.
  • Idempotency & Partial Refunds: Ensure external calls (payment gateways, ledgers) are strictly idempotent. The system must track cumulative refunds per order line-item and reject requests exceeding the original charge.
  • Crash Recovery: If a worker node crashes mid-execution, the workflow must resume from its last completed state without duplicate side effects.
  • Human-in-the-Loop: Support asynchronous pausing and resumption if an automated fraud threshold triggers manual review.
  • Status & Observability: Provide real-time queryable status for running and completed workflows.

Non-Functional Requirements & Scale

  • Scale: 10 million completed orders per day; refund rate is roughly 3% (~300,000 refund workflows per day, peak ~30 requests/second).
  • Durability: Zero tolerance for lost state transitions or duplicate payouts.
  • Latency: Automated workflows should finish within 3 seconds (excluding external payment gateway latency).

What to Cover

  1. Core Data Models: State representation for workflow instances, node execution states, dependency tracking, and financial idempotency keys.
  2. Architecture: Decoupling of workflow orchestrator, task queues, and worker execution.
  3. Reliability & Idempotency: Handling transient failures, external timeouts, crash recovery, and out-of-order execution.
  4. Concurrency & Locking: Preventing race conditions when two concurrent refund requests arrive for the exact same order.

Hints

Hint 1

Consider an event-sourcing or state-machine approach where each node transition writes an immutable event before dispatching the next task.

Hint 2

Think about how deterministic idempotency keys derived from the workflow ID, step ID, and attempt count can protect third-party gateway interactions.

Hint 3

To handle partial refund races on the same order, use distributed locks or pessimistic database-level locks on the parent order record before initiating the workflow.

Follow-up questions

  • How would you implement a compensation mechanism (Saga pattern) if a non-retryable step fails downstream after payment has already been captured or released?
  • How would you handle backpressure and dead-letter queues if a downstream third-party payment partner experiences a multi-hour outage?
  • How does your state machine scale if workflows need to sleep for several days awaiting manual support agent approval?