Katkat
Back

Design a Task Scheduler for Opaque Long-Running GPU Jobs ("Design Sora")

GoogleSoftware Engineer · Technical Screen
HardSoftware EngineerTechnical ScreenSystem Design

Last updated

This question tests your ability to design a distributed job scheduler and execution engine for long-running, compute-intensive GPU workloads. It evaluates distributed system fundamentals including multi-tenant fairness, gang scheduling, node failure recovery, and resource utilization.

Scenario

Design a distributed job scheduling and orchestration platform for a large-scale video generation service (similar to OpenAI Sora). Unlike standard microservice workloads that complete in milliseconds, video generation tasks are compute-heavy, opaque black-box jobs that run on specialized multi-GPU nodes (e.g., 8x H100 hosts) for anywhere from 2 to 45 minutes.

GPUs are extremely scarce and expensive, making utilization and fairness paramount. Your service must accept user generation prompts, assign work to available GPU infrastructure, manage execution, and reliably return generated video assets.

Requirements

Functional Requirements

  • Job Lifecycle API: Clients can submit jobs (with model parameters, prompt, and callback URL), poll job status, stream intermediate progress, and cancel running jobs.
  • Multi-Tenant Scheduling: Support multi-tier priority (e.g., Enterprise vs. Free tier) and fair-share scheduling to prevent noisy neighbors from starving other users.
  • Gang Scheduling: Support jobs that require tightly coupled multi-GPU allocation (e.g., all 8 GPUs on a single host or across multiple hosts) that must be allocated atomically.
  • Checkpointing & Resiliency: Gracefully handle worker node hardware crashes or thermal throttling mid-job without losing all progress.

Non-Functional Requirements

  • High GPU Utilization: Target > 85% cluster-wide GPU utilization.
  • Scalability: Manage a pool of 10,000+ GPUs serving 100,000 job requests per day.
  • Fault Tolerance: Worker failures or network partitions must not corrupt job state or cause permanent resource leaks (dangling allocations).

What Your Design Should Cover

  1. API & Data Model: Key API contracts and database schema tracking job state transitions (QUEUED, ASSIGNED, RUNNING, CHECKPOINTING, COMPLETED, FAILED).
  2. Architecture: Core components (API Gateway, Central Scheduler, Queueing Layer, Node/Host Agents, Artifact Storage).
  3. Scheduling Strategy: Algorithm for fair queuing (e.g., Dominant Resource Fairness or Weighted Deficit Round Robin), gang reservation, and backfilling.
  4. Fault Tolerance & Worker Management: Heartbeats, health checks, preemption, checkpoint storage, and cleanup protocols for stuck jobs.

Hints

Hint 1

Decouple admission control and fair queuing from the low-level machine-binding logic to avoid head-of-line blocking in the scheduler.

Hint 2

Think about how gang scheduling requires atomic all-or-nothing reservations to prevent distributed resource deadlocks.

Hint 3

Consider two-phase execution with periodic intermediate checkpointing to object storage so preempted or failed jobs can resume instead of restarting from scratch.

Follow-up questions

  • How would you detect and handle 'gray failures', such as GPU thermal throttling or silent memory corruption, that do not trigger a hard crash?
  • How does your scheduler implement backfilling to utilize idle GPU slots without delaying pending large gang-scheduled jobs?
  • If an enterprise tenant submits 5,000 jobs simultaneously, how do you mathematically guarantee fair allocation to smaller tenants while maintaining high overall cluster utilization?