Katkat
Back

Design the GPU Job Scheduler for a Text-to-Video Generation Service

OpenAISoftware Engineer · Technical Screen
HardSoftware EngineerTechnical ScreenSystem Design

Last updated

Design a high-throughput, fault-tolerant GPU job scheduler tailored for large-scale text-to-video generation workloads under sustained compute contention. The problem evaluates distributed resource allocation, multi-node gang scheduling, fairness guarantees, and preemption mechanics.

Scenario

You are tasked with designing the core GPU scheduling platform for a global text-to-video AI service. Generating video via diffusion and autoregressive video models requires multi-node, multi-GPU training/inference pipelines that consume significant compute, require high interconnect bandwidth (NVLink / InfiniBand), and run anywhere from 30 seconds (low-resolution interactive drafts) to 20 minutes (high-definition, 60fps final renders). Compute demand consistently outstrips capacity.

Requirements

Functional Requirements

  • Multi-Tenant Scheduling: Support distinct user tiers (e.g., interactive real-time previews, paid tier-1 production pipelines, and internal background model fine-tuning) with configurable SLAs.
  • Gang Scheduling: Video generation jobs require all allocated GPUs (e.g., 8 to 64 GPUs across multiple physical nodes) to start simultaneously; partial allocations cannot execute.
  • Preemption & Checkpointing: Lower-priority batch jobs must yield resources to interactive requests with minimal wasted compute.
  • Fair Sharing: Prevent any single organization or tenant from starving the cluster during burst traffic.

Non-Functional Requirements & Scale

  • Cluster Size: 16,000 GPUs arranged in 8-GPU nodes (2,000 nodes) interconnected via InfiniBand.
  • Job Volume: 50,000 job submissions per hour during peak periods.
  • Latency: Interactive draft scheduling decision latency must be under 200 ms; cluster state updates must reflect node/GPU health in near-real-time (< 1s).
  • High Availability: Scheduler must tolerate master node crashes and worker hardware failures without losing in-flight jobs.

What Your Design Should Cover

  1. API & Data Model: Endpoints for job submission, cancellation, and job state modeling (GPU request count, topology affinity, priority tier).
  2. Scheduler Architecture: Allocation engine (centralized vs. two-level), queueing model, and gang scheduling mechanics.
  3. Topology-Aware Placement: How the scheduler handles NUMA/rail-aligned InfiniBand and NVLink constraints.
  4. Fairness & Preemption Engine: Fair-share algorithms (e.g., Dominant Resource Fairness or Weighted Fair Queueing), preemption protocols, and checkpoint persistence.
  5. Failure Modes & Recovery: Handling stragglers, node heartbeats, and re-enqueueing aborted jobs.

Hints

Hint 1

Consider the constraints of gang scheduling across physical nodes; partial allocation causes deadlocks or idle GPU waste.

Hint 2

Topology awareness is critical: running an 8-GPU job across two nodes over standard Ethernet performs orders of magnitude worse than keeping it on one node or across dedicated InfiniBand rails.

Hint 3

Evaluate the trade-off between graceful preemption (waiting for the next diffusion step checkpoint) versus hard eviction with state replay.

Follow-up questions

  • How does your scheduler minimize GPU idle time (bubble time) between successive jobs running on the same node without reloading multi-gigabyte model weights into VRAM?
  • If a single GPU drops out due to an uncorrectable ECC error mid-render, how does the scheduler handle state recovery and re-assignment without restarting the full 20-minute job from scratch?
  • How would you adapt the scheduling policy if tenants submit jobs with elastic resource profiles (e.g., can run on 4, 8, or 16 GPUs with differing execution times)?