Monitor Devices Over an Unreliable Network

Scenario

You are tasked with designing a centralized monitoring service for a fleet of remote edge devices deployed worldwide. These devices operate over unpredictable, high-latency, and intermittently disconnected Internet connections (cellular, satellite, flaky broadband).

Because connections drop frequently, device telemetry often arrives in bursts, out of order, or after long delays. Devices may also reboot unexpectedly, reset their internal clocks, or retransmit duplicate packets. The system must provide operators with actionable visibility into fleet health while explicitly reflecting state uncertainty rather than making false binary claims (e.g., falsely asserting a device is dead when it is simply experiencing an uplink delay).

Requirements

Functional

• Telemetry Ingestion: Ingest regular heartbeats, resource metrics (CPU, memory, disk), and critical event logs from devices.
• Liveness & State Modeling: Compute the real-time operational status of each device (e.g., Healthy, Degraded, Unreachable / Stale, Offline). The status must represent confidence levels or time elapsed since last verified contact.
• Query & Dashboarding API: Allow operators to query the current state of any device and view recent status transition history.
• Alerting Engine: Trigger alerts when devices stay unreachable beyond configurable thresholds or report critical hardware/software errors.

Non-Functional

• Scale: Support 500,000 active devices reporting heartbeats every 15 seconds (~33,000 write QPS average, with bursts up to 100,000 QPS during regional reconnects).
• Clock Drift Tolerance: Device local timestamps cannot be trusted for absolute global ordering.
• Fault Tolerance: Loss of an ingestion node or temporary database unavailability must not corrupt device lifecycle state.

Key Design Challenges

1. Ambiguous Liveness: How to distinguish between an actual crash/power-off and a prolonged network partition or slow upstream queue.
2. Reboots & Deduplication: How to reliably order events and discard retransmissions when a device restarts, resetting TCP connections and potentially local sequence numbers.
3. Thundering Herd: Devices that reconnect after an outage will flush locally buffered telemetry all at once.

What Your Design Should Cover

• Agent-to-Server Protocol: Payload structure (boot IDs, sequence counters, client vs. server timestamps), transport protocol (MQTT vs. HTTP/gRPC), and buffering strategy on the device.
• Architecture: Ingestion tier, message bus, stream processing/state evaluator, and storage layer.
• State Machine: Specific states and transitions, including timeout windows and confidence decay.
• Data Model: Storage schema for current state snapshots and historical state transitions.