Katkat
Back

Design Batch Reboots for Machines

CoreweaveSite Reliability Engineer · Onsite
MediumSite Reliability EngineerOnsiteSystem Design

Last updated

This question tests the ability to design a distributed system that orchestrates machine reboots while maintaining cluster capacity, service availability, and operational safety.

Problem Statement

You are tasked with designing a system that manages the rolling reboot of 10,000 GPU-enabled nodes in a data center. The business requirement is to reboot machines to apply firmware and kernel updates without violating a 'Minimum Available Capacity' threshold.

Requirements

  1. Safety Constraints: Ensure at least 20% of the fleet remains online and healthy at all times.
  2. Orchestration: The system should handle concurrent reboots while avoiding mass outages.
  3. Health Checks: A machine must pass a pre-reboot check (e.g., draining workloads) and a post-reboot health check (e.g., verifying GPU availability) before it is marked as 'Ready'.
  4. Failure Handling: If a machine fails to come back online within a specific timeout, the system must halt and alert an operator.

Design Expectations

  • Architecture: Describe the high-level components of your reboot orchestrator (e.g., controller, state store, agent).
  • API Design: Define the necessary endpoints or command structures to initiate and track reboot tasks.
  • Data Model: How would you track the state of 10,000 machines across different phases (Pending, Draining, Rebooting, Verifying, Complete)?
  • Scalability/Reliability: How do you ensure the system is resilient to the orchestrator crashing? How do you prevent a 'thundering herd' of reboots?

Assumptions

  • The fleet is managed via an existing API that supports drain_node(node_id) and reboot_node(node_id).
  • There is an observability service that can verify machine health via check_health(node_id).

Hints

Hint 1

Consider using a state machine pattern to track the status of each node through the transition process.

Hint 2

Use a distributed lock manager or a centralized queue to ensure only a controlled number of reboots occur simultaneously.

Hint 3

Implement a 'check-point' logic where the system validates current cluster capacity before issuing any new reboot commands.

Follow-up questions

  • How would you modify your design if you had to perform reboots across multiple geographic regions simultaneously?
  • If a reboot fails, what automated recovery strategies would you implement before alerting a human?
  • How would you design the system to handle a priority override, where specific critical workloads need to be moved before a machine is rebooted?