Katkat
Back

Design a Human Workflow Distribution System

CoreweaveSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Design a distributed workflow allocation platform that routes tasks to human operators, tracks execution state via leases, and ensures fair queueing and fault tolerance.

Scenario

Design a human-in-the-loop workflow distribution system for an infrastructure operations team. The platform assigns specialized operational tasks (such as manual GPU host triage, edge-case hardware validation, and policy compliance verification) to human operators based on their skills, current capacity, and task priority.

The system must reliably manage the full task lifecycle—from ingestion to completion—while handling worker disconnections, lease expirations, and strict SLA deadlines.

Requirements

Functional Requirements

  • Task Ingestion & Prioritization: Systems submit tasks with attributes such as required skill sets, priority level, and maximum completion SLA.
  • Worker Matching & Distribution: Operators connect to the system, report their availability, and receive tasks matching their qualification levels (support push or pull model).
  • State Tracking & Leases: Tasks transition through explicit states (QUEUED, ASSIGNED, IN_PROGRESS, COMPLETED, FAILED, EXPIRED). A task assigned to an operator must be leased with periodic heartbeat renewals.
  • Failure & Timeout Handling: If an operator drops offline or fails to submit before lease expiration, the task must be automatically reclaimed and re-queued without state corruption.
  • Observability & Auditing: Complete audit trail of state transitions, time-to-assign, active handling duration, and operator performance metrics.

Non-Functional Requirements

  • Reliability: At-least-once task execution; no task may be lost due to network partitions or process crashes.
  • Fairness & Anti-Starvation: Low-priority tasks must not starve indefinitely, and work should be evenly distributed among eligible operators.
  • Consistency: Avoid race conditions where multiple operators work on the same task simultaneously.

Scale & Constraints

  • 10,000 concurrent active human operators globally.
  • 1,000,000 tasks generated per day, with peak ingestion of 100 tasks/second.
  • Task durations range from 30 seconds to 30 minutes.
  • System must support task assignment/pull latency under 200 ms at p99.

What Your Design Should Cover

  1. High-Level Architecture: Ingestion pipeline, queueing/scheduling tier, worker dispatch interface, and state store.
  2. Data Model & State Machine: Core entities (Task, Operator, Assignment/Lease) and state transition diagrams.
  3. Assignment Mechanism: Push vs. pull trade-offs, priority queues, and skill-based routing.
  4. Leasing & Heartbeats: Mechanics for lease renewal, zombie worker detection, and safe reassignment.

Hints

Hint 1

Consider whether a push-based dispatch or a pull-based worker queue model is better suited for varying operator availability and disparate task completion times.

Hint 2

Use a visibility timeout or lease pattern (similar to SQS or distributed locks with fencing tokens) to prevent duplicate assignments while ensuring automatic retries on worker dropouts.

Hint 3

To avoid starving lower-priority tasks, explore multi-level priority queues with aging mechanisms or weighted fair queueing.

Follow-up questions

  • How would you detect and handle a 'poison task' that repeatedly crashes the operator's interface or causes consecutive lease timeouts?
  • If a task requires multi-party consensus (e.g., dual verification where two operators must agree before marking it complete), how does the state machine and assignment logic evolve?
  • How would you implement task preemption if a critical, Sev-0 infrastructure task arrives while all qualified operators are already occupied with lower-priority tasks?