Schedule GPU Pods and Drain a Node
Together AISoftware Engineer · Technical ScreenLast updated
Implement a GPU cluster manager to schedule workloads using a deterministic best-fit policy and simulate draining a node to check if all its pods can be migrated without disrupting other nodes.
Problem Statement
You are tasked with building a deterministic GPU scheduler for a cluster manager. The cluster consists of several worker nodes, each with a fixed total number of GPUs. Workload requests are submitted as "pods", where each pod requests a specific integer number of GPUs that must be allocated entirely on a single node.
Implement the GPUClusterManager class with the following methods:
__init__(nodes: dict[str, int]): Initializes the cluster with a mapping ofnode_idto its total GPU capacity.schedule_pod(pod_id: str, gpus_needed: int) -> str | None: Schedules a pod onto an eligible node using a Best-Fit policy. An eligible node must have at leastgpus_neededavailable GPUs. Among all eligible nodes:- Choose the node with the minimum remaining GPUs that can still accommodate the pod (exact fits preferred).
- If there is a tie, select the node with the lexicographically smallest
node_id. If scheduled, deductgpus_neededfrom that node's available GPUs, record the placement, and return thenode_id. If no node can satisfy the request, returnNone(the cluster state remains unchanged).
can_drain(node_id: str) -> bool: Performs a dry-run check to determine ifnode_idcan be safely drained. Draining requires that all pods currently running onnode_idcan be reassigned to the other nodes in the cluster without evicting or moving workloads already running on those other nodes. To simulate the reassignment deterministically, pods onnode_idare rescheduled one by one in descending order of GPU requirement (ties broken bypod_idlexicographically). Each pod is placed using the same Best-Fit policy described above on the remaining nodes. ReturnTrueif every pod onnode_idcan be placed, orFalseotherwise. This check must not mutate the cluster's actual state.
Example
manager = GPUClusterManager({"node-1": 8, "node-2": 8, "node-3": 4})
# Schedule pods
manager.schedule_pod("p1", 4) # Returns "node-3" (exact fit: 4 remaining)
manager.schedule_pod("p2", 4) # Returns "node-1" (tie between node-1 and node-2; node-1 is smaller)
manager.schedule_pod("p3", 6) # Returns "node-2" (remaining 8 >= 6; node-1 only has 4)
# Node state:
# node-1: 4 used, 4 free (holds ["p2"])
# node-2: 6 used, 2 free (holds ["p3"])
# node-3: 4 used, 0 free (holds ["p1"])
manager.can_drain("node-1")
# "p2" needs 4 GPUs. Other nodes free: node-2 has 2, node-3 has 0.
# Cannot fit "p2". Returns False.
manager.schedule_pod("p4", 2) # Returns "node-2" (exact fit: 2 remaining)
manager.can_drain("node-3")
# "p1" needs 4 GPUs. node-1 has 4 free, node-2 has 0 free.
# "p1" fits on node-1. Returns True.
Constraints
1 <= len(nodes) <= 500- Total GPU capacity per node:
1 <= capacity <= 64 1 <= gpus_needed <= 64pod_idandnode_idare non-empty alphanumeric strings.- All
pod_idvalues passed toschedule_podare unique.
Hints
Hint 1
Track both total capacity and remaining capacity per node, along with a mapping of which pods reside on each node.
Hint 2
For Best-Fit, you can either iterate over all nodes or maintain a sorted/heap-based structure; given `N <= 500`, a linear scan filtering eligible nodes and finding the minimum tuple `(remaining_capacity, node_id)` is efficient and straightforward.
Hint 3
For `can_drain`, make a copy or lightweight snapshot of remaining capacities across all nodes except the drained node, then iterate through the drained node's pods sorted by `(-gpus, pod_id)`.
Follow-up questions
- Greedy descending Best-Fit is an approximation of bin packing. How would you solve `can_drain` if the requirement was strictly exact feasibility (NP-hard bin packing), and what bounds on pod count would make that feasible?
- How would you adapt `schedule_pod` to handle concurrent requests safely without turning the scheduler into a single-threaded bottleneck?
- If nodes also have constraints like NUMA socket locality or interconnect topology (e.g., NVLink domains of 8 GPUs), how would you represent node capacity beyond a single scalar?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.