Katkat
Back

Design a Video Generation Service with GPU Fleet Management

OpenAISoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenML System Design

Last updated

Tests the ability to design an end-to-end distributed ML inference system for compute-intensive video generation, focusing on GPU scheduling, job lifecycle management, fault tolerance, and cost efficiency.

Scenario

Design an asynchronous video generation platform (similar to OpenAI Sora) that converts user text and image prompts into high-fidelity video clips. Video diffusion models require massive compute (often spanning multi-GPU nodes for several minutes per generation), making scheduling, fault tolerance, and GPU fleet efficiency critical.

Requirements

Functional

  • Job Submission & Tracking: Clients submit generation requests specifying prompt, duration, resolution, and model version. Clients can poll status, stream progress, or register webhooks.
  • Content Moderation: Enforce two-stage moderation (pre-generation text/image analysis and post-generation safety checks on rendered frames).
  • Job Management: Support immediate job cancellation, bounded retries on transient infrastructure failures, and strictly idempotent artifact publication to object storage.
  • Versioning: Support simultaneous execution across multiple model checkpoints and inference runtime versions.

Non-Functional & Scale

  • Scale: 50,000 video requests per day (~2-5 requests/second average, peaking at 15 requests/second). Average generation takes 3 minutes on an 8-GPU node.
  • GPU Fleet: Manage a heterogeneous pool of 2,000–4,000 GPUs (e.g., NVIDIA H100, A100-80GB).
  • Reliability: Hardware faults (OOMs, silent data corruption, host reboots) must not cause lost user jobs or perpetual GPU hangs.
  • Cost Efficiency: High GPU utilization (>80%), dynamic autoscaling, and tenant rate/cost limiting.

What to Cover

  1. API & Job State Machine: Define the core client endpoints and a durable state machine (e.g., QUEUED, MODERATING, SCHEDULED, RUNNING, VALIDATING, SUCCEEDED, FAILED, CANCELLED).
  2. System Architecture: High-level topology separating the control plane, distributed scheduler, and data/compute plane.
  3. Scheduler & Fleet Management: Resource-aware placement (GPU memory, model weights pre-warmed on disk/VRAM), lease-based worker heartbeats, and worker draining.
  4. Checkpointing & Fault Tolerance: Strategies for mid-generation checkpointing across multi-step diffusion models and mitigating compute loss on worker eviction.
  5. Autoscaling & Cost Controls: Scale-up/scale-down policies handling slow GPU provisioning times, spot/preemptible instances, and user concurrency quotas.

Hints

Hint 1

Decouple fast API validations and text-based moderation from the long-running GPU scheduling queue to prevent API thread starvation.

Hint 2

Use a lease-and-heartbeat mechanism between the central scheduler and GPU workers, backed by distributed locks, to detect worker node crashes and reassign jobs without duplicates.

Hint 3

Incorporate cache locality into scheduler placement decisions so jobs requiring specific model weights land on nodes that already have those checkpoints cached in local NVMe storage.

Follow-up questions

  • How would you handle a 'thundering herd' problem where thousands of GPU workers attempt to download a newly deployed 50GB model weight file simultaneously?
  • How can you utilize spot or preemptible GPU instances effectively for multi-minute generation tasks without risking excessive duplicate computation?
  • If a user cancels a running job, how do you propagate the cancellation signal across all worker ranks in a multi-GPU distributed inference job immediately?