Katkat
Back

Design a Lab Machine Fleet Monitor: Payload Dispatch, Hang Detection, and Remote Reboot

MicrosoftSoftware Engineer · Onsite
EasySoftware EngineerOnsiteSystem Design

Last updated

Design a system that dispatches compute payloads to lab machines that poll for work and shows on a central dashboard whether each payload is downloading, running, completed, crashed, or hung. It tests agent design, hang detection, crash and hang logs, authorized reboot and power cycle, reruns after interruption, and refreshable machine inventory.

Design a system that lets users monitor a collection of lab machines that run compute loads.

  • Machines periodically ask the system for work. A work item identifies a compute payload that the machine must download, verify, start, and then monitor.
  • A central dashboard shows whether each payload is downloading, running, completed, crashed, or appears hung. For crashed and hung payloads, operators need useful logs.
  • An authorized operator may request a reboot or a power cycle of a machine.
  • A payload that is interrupted starts again from the beginning after the machine recovers.
  • Payload results must be available from the portal.
  • Users can see each machine's OS, CPU, and state. This information updates every 2 minutes, and a user can also force a refresh.

No production code or cloud-specific service names are required. Draw the main components as you discuss them.

Constraints and Clarifications

  • Inventory (OS, CPU, machine state) refreshes on a 2-minute cycle, plus on demand.
  • An interrupted payload is rerun from the start. It does not resume from a checkpoint.
  • Only authorized operators may reboot or power cycle a machine.
  • Assume you may install software of your own design on every lab machine.
  • No scale numbers were given. State any you assume.

Clarifying Questions

  • How many machines are there, how many payloads run on one machine at once, and how large are payloads, logs, and results?
  • Does a payload emit any progress or heartbeat signal, or must "appears hung" be inferred from outside the process? Is the hang threshold the same for every payload?
  • Can the system open connections to the machines, or can only the machines connect outward (for example, from behind a lab firewall)?
  • Do the machines have out-of-band management hardware, such as a baseboard management controller or a switched power unit, that can power cycle them when the operating system is unresponsive?
  • How quickly must a forced refresh show new data, and how long must logs and results be retained?
  • Who creates work items, and can a work item target a specific machine or any machine with a suitable OS and CPU?

Hints

Hint 1

Who can reach whom: Machines start every conversation by asking for work. Consider how a reboot request or a forced refresh reaches a machine between two of its check-ins, and what still works when the machine's own software is frozen.

Hint 2

Hung is not the same as silent: If the dashboard only notices that reports stopped, it cannot tell a hung payload from a crashed agent, a dead machine, or a network problem. Decide which component can observe each of those conditions.

Hint 3

Runs, not just work items: Because an interrupted payload starts over, one work item can run more than once. Think about how the status, logs, and results of an interrupted run stay separate from the run that replaces it.

Follow-up questions

  • An operator power cycles a machine while its agent is uploading a result. What does the dashboard show, and what keeps a partial result out of the portal?
  • How would you roll out a new version of the machine agent across the fleet without disrupting running payloads?
  • How do you make sure a downloaded payload is the one the work item intended, and what happens if verification keeps failing on one machine?
  • Hundreds of users press force refresh on the same machine at once. How do you protect the machine and the backend?