Design Command Dispatch and Telemetry Reconciliation for Unreliable Devices

Scenario

Your organization manages an active fleet of 5 million edge devices (such as robotic systems, remote sensor gateways, and local inference nodes). Operators and automated workflows issue imperative commands (e.g., trigger diagnostic routine, swap active model weights, restart subsystem, emergency stop) to target devices. Devices frequently disconnect, operate offline for hours or days, and reconnect with high packet loss and high latency.

Design the end-to-end command dispatch pipeline and telemetry reconciliation engine that coordinates command execution, tracks per-device progress, and maintains consistent fleet state.

Requirements

Functional Requirements

• Targeting & Scheduling: Dispatch commands to explicit device IDs or dynamic tags (e.g., model=v2-arm, region=us-east). Support immediate execution, scheduled execution (future timestamp), and expiration (Time-to-Live).
• Command Lifecycle Tracking: Operators must be able to query the lifecycle state of a command per device: QUEUED, DISPATCHED, ACKNOWLEDGED, EXECUTING, COMPLETED, FAILED, or EXPIRED.
• Telemetry Reconciliation: Process incoming device status reports, heartbeats, and execution telemetry to update command status. Must handle out-of-order, delayed, or duplicated reports.
• Safety & Cancellation: Operators can cancel in-flight or queued commands. Devices must not execute stale or revoked commands upon reconnecting.

Non-Functional Requirements

• Scale: 5 million registered devices; up to 1 million concurrently connected. Average telemetry rate of 1 event per device every 30 seconds (~33,000 telemetry events/sec baseline, with reconnect bursts up to 200,000 events/sec). Command volume of up to 10,000 dispatch requests/sec.
• Correctness & Idempotency: Exactly-once semantic execution at the application level; devices must never execute the same action twice.
• Durability: Zero command loss for commands accepted by the system within their TTL.

Deliverables

1. API & Data Model: Core APIs for command submission, cancellation, and device status queries. Data schema for commands, device command queues, and execution logs.
2. System Architecture: Component diagram detailing command submission, targeting resolution, device connectivity/gateway layer, and telemetry ingestion pipeline.
3. Reconciliation Strategy: Explain how the system handles late-arriving telemetry, clock drift, and conflicts between server-side state and edge reality.
4. Failure Modes & Safety: Strategies for handling network partitions during command execution, fleet-wide reconnect storms, and zombie command execution.