Scale Mixed I/O and LLM Workflow Execution

Design the capacity, queueing, worker isolation, and storage layers of a multi-tenant workflow system whose DAGs mix fast I/O tasks with much slower LLM tasks.

Reported Workload and Constraints

The run count multiplied by the average task count implies 10 million, not 100 million, daily task executions. Preserve this discrepancy as a sizing question. A state-transition record also need not correspond one-to-one with a task execution. These figures are interview inputs, not verified measurements of a deployed system.

Clarifying Questions to Ask

• Which daily execution count is authoritative, and how many state events does one attempt generate?
• Do task classes have the same proportions at peak as across the day?
• What retention and query patterns apply to definitions, live run state, task history, logs, and large intermediate objects?
• Which fairness or latency guarantees apply per tenant, and can unused capacity be borrowed?

Part 1 — Size and Separate Fast and Slow Work

Estimate average dispatch rates under both stated daily volumes and explain how runtime differences affect in-flight concurrency. Design queues and worker pools that prevent slow LLM calls from exhausting the slots needed by fast I/O tasks.

What This Part Should Cover

• Correct arithmetic and explicit uncertainty rather than silently choosing a workload figure.
• Separate admission, concurrency, and backpressure for task classes.
• Capacity estimates that distinguish averages, bursts, and provider-limited throughput.

Part 2 — Store Definitions, State, Logs, and Payloads

Describe records, indexes, partition keys, retention, and storage tiers for the reported scale. Explain how to query a run's current progress without scanning its entire historical log or carrying large document bodies in the metadata database.

What This Part Should Cover

• Immutable definition versions and durable run/task/attempt identities.
• Hot state versus append-only history and large object payloads.
• Partitioning, bounded indexes, retention, and recovery-safe archival.

Part 3 — Protect Tenants from Bursty Neighbors

A tenant submits hundreds of thousands of runs at once. Explain admission and scheduling controls that protect other tenants while allowing useful work to continue.

What This Part Should Cover

• Tenant limits on submissions, queued work, running tasks, and scarce downstream resources.
• Fair dispatch within each task class and controlled borrowing of spare capacity.
• Metrics that expose class-specific and tenant-specific starvation.