Design a Sora-Style Video Generation Service with Fair Queueing and Cancellation

Design the backend of a text-to-video generation product in the style of Sora. A user submits a text prompt with generation settings, the request becomes a long-running generation task that runs on a GPU, and when it finishes the user retrieves the video.

The design has one simplifying constraint: each task runs on exactly one GPU. A single task is never split across several GPUs, so the scheduler never has to find several free GPUs at once for the same task.

GPU capacity is the scarce resource, and many users compete for it. The discussion should concentrate on two areas: how tasks from different users are queued fairly for the GPU fleet, and how a user cancels a task that is waiting or already running.

Constraints and Clarifications

• One task uses exactly one GPU for its whole run.
• Demand can exceed GPU capacity, so tasks wait in a queue before they run.
• Cancellation requests are expected to be infrequent compared with submissions.
• No latency, throughput or fleet-size targets are given; state the assumptions your design depends on.

Clarifying Questions

• Who must be treated fairly: individual users, organizations, or subscription tiers with different shares?
• Can a task's GPU time be estimated from its settings (for example resolution and clip length) before it runs?
• Does a GPU run one task at a time, and are all GPUs in the fleet interchangeable?
• Can a running task be paused and resumed later, or can it only be stopped?
• How do users learn that a video is ready: by polling, by a push notification, or both?
• Is a user charged for the GPU time a task used before it was canceled?

Part 1 — Architecture and task lifecycle

Describe the end-to-end design: the user-facing API, the states a task passes through from submission to a finished video, how queued tasks are handed to GPU workers, and where prompts, task state and generated videos are stored.

What This Part Should Cover

• A task state machine that includes the states a cancellation can interrupt.
• The dispatch mechanism between the queue and the GPU workers, and how a worker that dies mid-task is detected and its task recovered.
• Separate storage for task metadata and for large video outputs, and how the finished video reaches the user.

Part 2 — Fair queueing

Some users submit far more tasks than others, and tasks differ widely in how much GPU time they need. Propose several ways to share the GPU fleet fairly among users, compare their trade-offs, and choose one.

Clarifying Questions for this Part

• When only one user has queued tasks, may that user take the whole fleet?
• Should a user who has been idle for a while get extra priority when they return?

What This Part Should Cover

• At least two distinct fair-queueing approaches, with the concrete weakness of each.
• Handling of variable task cost, of bursts, and of users returning after being idle.
• Where the policy runs, and what happens to its state if the scheduler restarts.

Part 3 — Cancellation

A user can cancel a task while it is queued, while it is being handed to a GPU, or while it is running. Design cancellation for each of these cases and compare approaches. In particular, evaluate this proposal: because each task holds at most one GPU and cancellations are infrequent, the cancel API can simply make a direct RPC or HTTP call to the GPU worker running the task. Say when that is sufficient and where it stops being extensible.

What This Part Should Cover

• Cancellation behavior in each task state, including the dispatch race and a task that has just finished.
• The direct worker call weighed against at least one alternative, argued from the one-GPU-per-task constraint and the low cancel volume.
• Delivery guarantees: idempotent cancel requests, an unreachable worker, and the point at which the GPU returns to the pool.