Katkat
Back

Design a Scheduler for ML Training and Batch Inference Jobs

NetflixSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

This question tests the ability to design a distributed job scheduler capable of handling heterogeneous ML workloads with varying priority, resource requirements, and fault-tolerance constraints.

Problem Statement

Design a distributed job scheduling system that manages compute resources for ML training (long-running, high memory/GPU requirements), batch inference (high throughput, predictable latency), and evaluation (low priority, sporadic) jobs.

Functional Requirements

  • Job Submission: Users submit jobs with resource requirements (CPU, RAM, GPU) and priority levels.
  • Scheduling: The system must assign jobs to available compute nodes based on resource availability and fairness policies.
  • Multi-tenancy: Support isolation between different teams to prevent a single job from starving others.
  • Fault Tolerance: Handle node failures and automatic job retries.

Non-Functional Requirements

  • Scalability: Support thousands of nodes and tens of thousands of concurrent jobs.
  • Reliability: Minimal downtime; job state must persist across scheduler restarts.
  • Fairness: Implement a mechanism to prevent starvation of lower-priority jobs while ensuring high-priority jobs meet SLA targets.

Expectations

Your design should cover:

  1. The API and data model for job submission and resource tracking.
  2. The architecture of the scheduler (e.g., centralized vs. distributed).
  3. Handling of resource preemption and oversubscription.
  4. Mechanisms for job monitoring and health checking.

Include a high-level diagram or textual description of the workflow from job submission to execution.

Hints

Hint 1

Consider using a queue-based approach for pending jobs and a heart-beating mechanism for resource monitoring on the worker nodes.

Hint 2

Think about how to use a metadata store (like etcd or Zookeeper) to track cluster state and ensure high availability of the scheduler itself.

Hint 3

Address the fairness issue by using resource quotas per team and a weighted round-robin or DRF (Dominant Resource Fairness) algorithm for scheduling.

Follow-up questions

  • How would you implement resource preemption if a high-priority training job arrives and the cluster is at 100% capacity?
  • How would you handle the challenge of bin-packing jobs efficiently to minimize wasted resources on nodes?
  • If the scheduler becomes a bottleneck at scale, how would you evolve the architecture to partition the load?