Katkat
Back

Design real-time per-status device counts

VerkadaSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

This question tests the ability to design a distributed system capable of aggregating real-time device heartbeat events into status counts while handling event-time latency and fault tolerance.

Problem: Real-Time Device Status Aggregator

You are tasked with designing a service that tracks the status of millions of IoT devices. Each device sends a heartbeat event every 10 seconds. An event includes a device_id, a status (e.g., 'online', 'offline', 'error'), and a timestamp generated by the device.

The system must output the count of devices per status aggregated in 1-minute windows (e.g., how many devices reported 'online' between 12:00:00 and 12:01:00).

Requirements:

  • Scalability: Handle 10 million devices sending heartbeats every 10 seconds.
  • Accuracy: The system must handle late-arriving events (events arriving at the server minutes after the device generated them) and out-of-order events.
  • Fault Tolerance: Ensure no data is lost if a processing node fails.
  • Latency: Provide an end-to-end latency of less than 30 seconds for the aggregated counts.

Your design should cover:

  1. Data Pipeline: How events flow from the device to the final storage.
  2. Aggregation Strategy: How you handle windows, state management, and late data.
  3. Failure Handling: How to ensure consistency if a server dies during aggregation.
  4. Scalability: How the system partitions data to handle the throughput.

Hints

Hint 1

Consider using a distributed message queue like Kafka to buffer incoming events and provide a unified log for multiple consumers.

Hint 2

Use a stream processing framework (like Flink or Spark Streaming) that supports 'watermarks' to handle late-arriving data and event-time processing.

Hint 3

Implement persistent storage for stateful aggregations (e.g., RocksDB or a distributed key-value store) to ensure the system can recover progress after a crash.

Follow-up questions

  • How would you handle a sudden 'thundering herd' of devices reconnecting after a network partition?
  • If the business needs to verify the status of a specific device at any time, how would you store the 'latest state' efficiently versus the windowed aggregation?
  • How would you modify the system if we needed to trigger an alert only when the number of 'error' status devices exceeds a threshold for three consecutive windows?