Katkat
Back

Design the Control Plane for a Compute Cluster: Host Monitoring + Job Dispatch

NVIDIASoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

Design a distributed control plane for a large compute cluster that ingests high-frequency host health and resource heartbeats while scheduling and dispatching jobs onto healthy hosts. The problem evaluates distributed state management, data modeling under write-heavy loads, failure detection, and high-availability coordinator design.

Scenario

You are tasked with designing the control plane for a dedicated compute cluster hosting accelerated workloads (e.g., GPU/accelerator training and inference tasks). The control plane is responsible for continuously monitoring cluster health, maintaining up-to-date node resource availability, and dispatching submitted compute jobs to suitable, healthy nodes.

Requirements

Functional Requirements

  • Node Heartbeating & Telemetry: Host agents on each worker node periodically report heartbeat, node health, and granular resource utilization (GPU memory, GPU compute usage, CPU, RAM).
  • Job Submission & Dispatch: Clients submit batch compute jobs requiring specific resource footprints (e.g., N GPUs of a specific model, CPU/RAM minimums). The control plane selects capable, healthy worker nodes and dispatches tasks.
  • Failure Detection & Recovery: If a node misses heartbeats beyond a threshold, mark it degraded/unhealthy, fail any active tasks assigned to it, and trigger rescheduling.
  • State Querying: An API for operators to inspect cluster-wide host health, aggregate resource utilization, and running job statuses.

Non-Functional Requirements

  • Scalability: Support 50,000 physical hosts and up to 500,000 concurrent containers/jobs.
  • Latency: Heartbeats reported every 5 seconds. Dispatch latency for queued jobs under 500ms.
  • Availability & Consistency: High availability for the control plane (99.99%). Prevent split-brain scheduling (a node must not be double-assigned conflicting workloads).
  • Fault Tolerance: Graceful failover of control plane leaders without dropping cluster state or orphaning running jobs.

Scale Estimates

  • 50,000 nodes sending heartbeats every 5 seconds = 10,000 heartbeats/sec write throughput.
  • Peak job submissions: ~500 jobs/sec.

Key Deliverables

  1. High-Level Architecture: Control plane components (API gateway, scheduler, state store, heartbeat ingestion pipeline).
  2. Data Model & Storage Strategy: Choice of primary transactional datastore vs. ephemeral in-memory state for heartbeats.
  3. Failure Detection Mechanism: Dead-man switches, lease mechanisms, or heartbeat timeout windows.
  4. Concurrency & Scheduling Safety: How to guarantee safe assignment without race conditions when multiple schedulers operate concurrently.
  5. Deep Dives & Trade-offs: Handling network partitions, leader election, and coordinator recovery.

Hints

Hint 1

Differentiate between durable metadata (job specs, cluster topology) and ephemeral, high-churn telemetry (heartbeats, live utilization) to avoid overwhelming a transactional database.

Hint 2

Use distributed leases or optimistic locking with resource versioning to avoid double-allocating a host's GPUs across concurrent schedulers.

Hint 3

Decouple heartbeat ingestion via an append-only in-memory cache or message bus from the scheduler evaluation loop to maintain low dispatch latency.

Follow-up questions

  • How does your design prevent a 'thundering herd' of rescheduling events if a rack-level switch momentarily drops heartbeats for 500 nodes?
  • If a scheduled node loses connectivity with the control plane but continues executing its assigned GPU workload, how do you prevent split-brain execution when reassigning the job?
  • How would you extend this system to support multi-node gang scheduling (all N nodes must be acquired simultaneously) without deadlocking concurrent jobs?