Katkat
Back

Schedule CPU and GPU Video Processing

NuroSoftware Engineer · Onsite
HardSoftware EngineerOnsiteSystem Design

Last updated

Design a distributed heterogeneous batch-processing system to schedule and execute multi-stage video pipelines across CPU and GPU worker clusters with fault tolerance, backpressure, and cost-efficient resource management.

Scenario

Autonomous vehicle fleets record thousands of driving logs and camera feeds daily. Design an ingestion and processing pipeline that handles 1,000,000 raw video clips per week. Each video must sequentially pass through three distinct processing phases:

  1. Ingestion & Preprocessing (CPU): Decode video stream, downsample, and extract keyframes.
  2. Model Inference (GPU): Run heavy 3D object detection and perception models on extracted frames.
  3. Post-Processing & Packaging (CPU): Fuse model detections with sensor telemetry, generate compressed summary artifacts, and write to long-term storage.

Requirements

Functional

  • Ingest, schedule, and orchestrate execution of video DAGs (Directed Acyclic Graphs) across heterogeneous compute pools.
  • Support dynamic worker assignment and priority queues (e.g., triage critical disengagements ahead of routine mapping logs).
  • Track pipeline state, log progress, and support idempotent stage-level retries without reprocessing completed upstream stages.

Non-Functional

  • Efficiency & Cost: Maximize GPU utilization (>85%) to minimize cloud compute cost; prevent expensive GPU nodes from idling on upstream CPU bottlenecks.
  • Scalability: Handle 1,000,000 videos/week with variable durations (average 5 minutes, 1080p at 30 fps).
  • Resilience: Gracefully handle node failures (including spot/preemptible instances) and network partitions.
  • Storage Limits: Prevent disk exhaustion from uncompressed intermediate frame dumps.

Scale & Capacity Assumptions

  • Ingestion rate: ~1.65 videos/sec sustained average, peaking at 5-10 videos/sec.
  • Average video: 5 minutes long, encoded H.264 (~500 MB).
  • Stage durations (single video benchmark):
    • Stage 1 (CPU): 3 minutes on 4 vCPUs.
    • Stage 2 (GPU): 1 minute on 1 modern GPU (e.g., NVIDIA A10G/T4).
    • Stage 3 (CPU): 1 minute on 2 vCPUs.

What to Cover

  1. Capacity Estimation: Calculate required CPU cores, GPU units, and memory/network bandwidth.
  2. System Architecture: Scheduler design, task queues, worker coordination, and metadata storage.
  3. Intermediate Data & Backpressure: How to transfer data between CPU and GPU stages without overwhelming local disk or network I/O.
  4. Failure Recovery & State Management: Checkpointing, idempotency, and handling worker preemption.

Hints

Hint 1

Balance your worker pool sizing based on the execution duration ratio (3 CPU min : 1 GPU min : 1 CPU min) to avoid pipeline starvation or excessive intermediate buffering.

Hint 2

Uncompressed video frames consume tens of gigabytes per clip; consider streaming frame batches directly into GPU memory via shared memory or local NVMe scratch disks rather than dumping them to object storage.

Hint 3

Decouple CPU and GPU stages using separate task queues with explicit capacity-aware backpressure limits (e.g., pausing Stage 1 workers when the Stage 2 queue depth exceeds a threshold).

Follow-up questions

  • How would you adapt the scheduling strategy if videos vary drastically in duration from 30 seconds to 2 hours?
  • How can you utilize cloud spot/preemptible instances for the GPU stage without incurring massive re-work penalties when instances get terminated?
  • If inference requires batched frames across multiple videos to achieve optimal GPU throughput, how would you design the batching and aggregation layer?