Katkat
Back

Design a Text-to-Video Generation System

OpenAISoftware Engineer · Onsite
HardSoftware EngineerOnsiteML System Design

Last updated

This question evaluates a candidate's competency in designing scalable ML inference platforms, including GPU resource management and scheduling, asynchronous job orchestration and state-machine-driven job lifecycles, failure classification and recovery, artifact storage, rate limiting, and monitoring for quality and latency.

Design a Sora-like text-to-video generation platform.

Users submit a text prompt, optional generation settings (duration, resolution, fps, seed, model variant), and possibly optional conditioning media such as an init image or reference clip. The system generates a short video and returns a downloadable result once the job is complete. Because a single clip takes seconds-to-minutes of GPU time, the system is inherently asynchronous: a submit returns a job handle immediately, and the user polls or receives a webhook when the result is ready.

Your design should cover the user-facing API and job lifecycle, the high-level service architecture, how GPU inference workers are scheduled, how the system handles unstable workers / crashes / retries / partial failures, how intermediate and final artifacts are stored, how safety / rate limits / quotas are enforced, and how quality / latency / reliability are monitored. The hardest, most-probed areas are (a) the job lifecycle / state machine and (b) failure handling when workers are unstable — go deep on both.

Constraints & Assumptions

State your own numbers in the interview — the figures below are illustrative, not benchmarks.

  • A single job is a short clip (e.g. 5–10 s at up to 720p, 24 fps), taking on the order of tens of seconds to a few minutes of GPU wall-time on one accelerator — 3–6 orders of magnitude slower than a typical web request.
  • The scarce, expensive resource is the GPU accelerator. API, queue, and database compute are cheap by comparison; the design optimizes for GPU utilization and fairness, not request throughput.
  • Jobs are long-running and stateful mid-flight. Worker crashes, OOMs, and spot-instance preemptions are routine, not exceptional — fault tolerance is a first-class requirement.
  • Output is regulated content: both the prompt and the generated frames must pass safety/moderation, and outputs typically require watermarking / provenance.
  • Assume a target sustained throughput (e.g. ~1 job/s) and multiple user tiers (free / paid / interactive preview) with different priority.
  • The system is fully async: every job must reach a terminal state and never silently hang.

Clarifying Questions to Ask

  • What clip lengths, resolutions, and frame rates must we support, and which of those are user-tunable vs fixed per tier?
  • What is the target throughput and the latency SLA (queue wait vs total time), and how does it differ across free / paid / preview tiers?
  • What conditioning inputs are in scope (text only, init image, mask, reference clip, audio)?
  • What are the safety/policy requirements (banned categories, real-person likeness, watermarking, provenance, audit retention)?
  • What is the retention policy for generated videos and intermediates, and are there data-residency or erasure (GDPR) obligations?
  • Is billing per-GPU-second, per-job, or subscription — and must we guarantee no double-billing on retries?

Hints

Hint 1

Where to start: Anchor the whole design on the **inference profile**: GPU-bound, long-running, expensive, failure-prone. That single observation forces an **async, queue-backed** architecture where the database is the source of truth for job state and the GPU is the scarce resource you schedule for fairness and utilization, not request throughput.

Hint 2

Job lifecycle: Draw an explicit **state machine** with terminal states (e.g. `COMPLETE`, `FAILED_PERMANENT`, `CANCELED`, `TIMED_OUT`) that never transition out, and give every `RUNNING` job a deadline so a wedged worker can't hang forever. Make `POST /videos` **idempotent** via an `Idempotency-Key` so a client retry on timeout never double-creates (and double-bills) a GPU job.

Hint 3

Failure model: Pick a single mechanism that lets the database decide who owns a job and when it has died, and that survives a worker that disappears and later comes back. Think about what damage a partitioned worker that reconnects could do to a job already retried elsewhere — and what property each result write would need to carry so that a stale, returning attempt can never win over the one the system already trusts.

Hint 4

Classify failures: Don't model "a job failed" as one thing. Sort failures by what the *right reaction* is — would retrying as-is have any chance of succeeding, would it only succeed under different conditions, or is it guaranteed to fail again? Map each class to retry / retry-differently / give-up, and ask what intermediate state a long job could persist so that a recovered attempt resumes cheaply instead of restarting from zero.

Hint 5

Scheduling & batching: The GPU is what you're optimizing for, so reason about what makes it idle or unfair: cold model-weight loads, one user monopolizing the pool, and small kernels. Think about which jobs you'd want to land on the *same* worker (weight affinity), how you'd stop a flood of free-tier work from starving paid traffic (priority + fair queuing), and the latency cost you pay when you micro-batch jobs together to fill the GPU.

Follow-up questions

  • A worker is network-partitioned, declared dead, its job retried and completed elsewhere — then the original worker reconnects and tries to publish its result. Walk through exactly how your design prevents a double publish or double bill.
  • 80% of a 90 s job completes and then the spot instance is preempted. How do you avoid redoing the expensive work, and what are the limits of resuming a non-deterministic GPU computation?
  • Inference finished successfully but the post-processing (encode / upscale / watermark) stage crashed. What do you re-run, what must you *not* re-run, and why?
  • A single prompt deterministically crashes whatever worker picks it up (a "poison pill"). How does the system keep this from taking down the fleet, and how do you bill it?
  • Your GPU utilization drops to 60% while queues are full and wait times climb. What are the likely causes, and how does your monitoring let you localize the bug to scheduling/placement vs the model vs hardware?