Katkat
Back

Design a Multi-Service Workflow Engine

DoorDashSoftware Engineer · Onsite
HardSoftware EngineerOnsiteSystem Design

Last updated

Design a resilient, distributed workflow orchestration engine capable of executing multi-step business processes across heterogeneous services with strict guarantees around durability, idempotency, and failure recovery.

Scenario

At DoorDash, critical business flows—such as order fulfillment, merchant onboarding, and driver payouts—span multiple microservices (e.g., Payment, Inventory, Dispatch, Notification). These workflows require strict execution guarantees: steps must execute in order, handle transient network blips, support rollback/compensation on terminal failures, and never duplicate non-idempotent real-world actions.

You are tasked with designing a workflow engine that orchestrates multi-service, asynchronous workflows reliably even under process crashes, network partitions, and host restarts.

Requirements

Functional Requirements

  • Workflow Definition: Support defining workflows as directed acyclic graphs (DAGs) or state machines with branching, parallel steps, and dependencies.
  • Task Dispatch & Execution: Coordinate execution across separate worker services via task queues or RPC/HTTP.
  • Failure Handling & Retries: Configurable exponential backoff retries per step, task timeouts, and automated compensation steps (Saga pattern) upon terminal failure.
  • Idempotency: Prevent duplicate executions of critical tasks (e.g., double-charging cards or duplicate dispatch notifications).
  • Observability & Inspection: Expose APIs to inspect workflow progress, step logs, and trigger manual pause/resume/replay.

Non-Functional Requirements

  • Durability: A crash of any engine node or worker must not corrupt or lose the execution state of in-flight workflows.
  • Scale: Handle 50,000 concurrently running workflows, with peak throughput of 10,000 state transitions per second.
  • Low Overhead: Internal scheduling latency between dependent steps should be under 100ms at p99.
  • Extensibility: Workers can be written in any language and register for specific task types.

What Your Design Should Cover

  1. High-Level Architecture: Core components (Scheduler/Engine, State Store, Task Queues, Worker Agents) and their interactions.
  2. Data Model & Execution State: How workflow definitions, run instances, and task state transitions are persisted (e.g., event sourcing vs. state machines).
  3. Worker Coordination: Mechanism for dispatching tasks to workers, heartbeat management, and dead-letter handling.
  4. Failure Recovery: Step-by-step recovery flow when the orchestrator crashes mid-workflow, or when a worker dies mid-task.
  5. Idempotency Strategy: How deduplication keys and task lease tokens ensure side effects are executed safely.

Hints

Hint 1

Consider whether an event-sourced history log (like Temporal or Cadence) or a mutable state-machine table better satisfies durability and auditability requirements at this scale.

Hint 2

Think about using lease tokens or fencing tokens with short TTLs and heartbeat pings to detect dead workers without triggering duplicate work.

Hint 3

Decide whether your architecture relies on central push orchestration or worker-pulled queues, and evaluate how backpressure is handled under load.

Follow-up questions

  • How would you prevent a 'zombie' worker (one that experienced a long GC pause or network partition) from writing stale results after its lease has already been reassigned?
  • If a workflow requires deterministic code replay to rebuild state, how do you handle versioning when a workflow definition changes while existing instances are still running?
  • How would you partition the database and task queues to scale the system horizontally without introducing cross-partition distributed transactions?