Katkat
Back

Monitor 100,000 GPU Hosts and Trigger Repair Workflows

OracleSite Reliability Engineer · Onsite
EasySite Reliability EngineerOnsiteSystem Design

Last updated

This question tests the ability to design a distributed telemetry collection and automated incident remediation system for a large-scale hardware fleet.

System Design: GPU Fleet Monitoring and Repair Service

You are tasked with designing an automated health monitoring and repair system for a fleet of 100,000 GPU-enabled hosts. Each host performs intensive AI/ML workloads. Your system must detect GPU hardware failures (e.g., ECC errors, thermal throttling, hang detection) and trigger automated repair workflows (e.g., resetting drivers, re-imaging the OS, or flagging for physical replacement).

Requirements

Functional Requirements:

  1. Telemetry Ingestion: Collect health metrics from all 100,000 hosts at a rate of 10Hz.
  2. State Management: Track the status of each GPU (Healthy, Degraded, Repairing, Failed).
  3. Automated Recovery: Execute a state-machine-driven repair workflow when a failure is detected.
  4. Blast Radius Control: Limit the number of concurrent repairs to prevent cascading service capacity loss.

Non-Functional Requirements:

  1. Scalability: Handle high-throughput metric ingestion and distributed workflow coordination.
  2. Idempotency: Ensure that repair workflows can be safely retried without side effects.
  3. Observability: Provide a global dashboard showing real-time fleet health and repair progress.

Scope of Response

  • Describe the high-level architecture (components, data flow, storage choices).
  • Design the ingestion pipeline to handle the volume of telemetry data.
  • Explain how you would implement the "guardrails" to ensure that automated repairs do not take down too many nodes simultaneously.
  • Define how you would handle escalation if an automated repair fails multiple times.

Hints

Hint 1

Consider using a distributed message queue like Kafka to decouple telemetry ingestion from processing and state updates.

Hint 2

Think of the repair workflow as a state machine where status transitions are stored in a highly available, consistent database like etcd or Zookeeper.

Hint 3

Implement a token-bucket or semaphore pattern to throttle the number of concurrent active repair tasks across the entire cluster.

Follow-up questions

  • How would you detect 'silent' failures where the GPU appears healthy to the OS but produces incorrect computation results?
  • If you need to perform a firmware update across the fleet, how would you design the deployment strategy to minimize downtime?
  • What metrics would you monitor to ensure the monitoring system itself is not failing?