Katkat
Back

Design GPU Scheduling for a Video Generation Platform

OpenAISoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

This question tests your ability to design a distributed job scheduling system that manages heterogeneous GPU resources for computationally expensive, variable-length video generation tasks.

Design a GPU Scheduling System for Video Generation

Video generation models (e.g., Diffusion-based transformers) are resource-intensive, requiring high-memory GPUs and varying inference times based on prompt complexity and video duration. You are tasked with designing a scheduling service that manages a cluster of hundreds of heterogeneous GPU nodes (e.g., A100s, H100s) to process a high-throughput stream of user requests.

Requirements

  • Functional:
    • Accept video generation requests with different priorities (e.g., "Pro" subscribers vs. "Free" users).
    • Efficiently match jobs to available GPU resources considering memory requirements and expected latency.
    • Support task preemption for high-priority jobs.
  • Non-Functional:
    • Maximize GPU utilization while minimizing queuing latency.
    • High availability: The scheduler should be resilient to single-node failures.
    • Scalability: The system must handle thousands of concurrent requests during peak traffic.

Design Expectations

Your solution should cover:

  1. API Design: Define the interface for job submission and status tracking.
  2. Architecture: Describe the interaction between the request orchestrator, the scheduling logic, and the GPU worker nodes.
  3. Data Model: How will you track GPU health, capabilities, and job queue state?
  4. Trade-offs: Explain how you balance "fairness" (preventing starvation of free users) with "throughput" (packing the cluster optimally).

Assumptions

  • Inference time is highly variable (5 seconds to 2 minutes).
  • GPUs have fixed VRAM capacities; models may require specific VRAM thresholds to run.

Hints

Hint 1

Consider implementing a tiered priority queue where different GPU resource pools are reserved for different subscription tiers.

Hint 2

Think about how to use a sidecar pattern or an agent on each GPU node to provide real-time telemetry (VRAM usage, temperature, current job) to the scheduler.

Hint 3

Address the cold-start problem: how will the system handle the time required to load large model weights onto the GPU memory?

Follow-up questions

  • How would you handle a sudden 10x spike in traffic, and how would this impact your auto-scaling strategy for GPU nodes?
  • If a long-running job is preempted, what mechanisms are needed to ensure the user doesn't lose progress or incur unnecessary costs?
  • How would you integrate an intelligent batching mechanism to run multiple smaller inference tasks on a single large GPU simultaneously?