Katkat
Back

Design Real-Time Electricity Monitoring and Policy Control

OpenAISoftware Engineer · Technical Screen
HardSoftware EngineerTechnical ScreenSystem Design

Last updated

Design a distributed real-time electricity monitoring and policy actuation system that ingests high-throughput telemetry, evaluates dynamic load-shedding policies with hysteresis, and reliably issues auditable commands to edge devices.

Scenario

Modern power-intensive facilities (such as AI data centers or smart industrial grids) require fine-grained power management to avoid exceeding substation quotas, prevent breaker trips, and participate in grid demand-response programs.

You are tasked with designing a real-time electricity monitoring and policy control platform. The platform continuously ingests power telemetry from millions of distributed smart meters and compute units, evaluates complex power policies in real time, and issues actuation commands (e.g., throttle compute nodes, shed HVAC load, switch to battery backup).

Requirements

Functional Requirements

  • Telemetry Ingestion: Continuously ingest power readings (voltage, active power, cumulative kWh, timestamp) from connected devices.
  • Stateful Policy Engine: Evaluate aggregate and per-device limits (e.g., "if datacenter rack power exceeds 35 kW for > 15s, shed lower-priority workloads"). Implement hysteresis to prevent "chattering" (oscillating between load shed and restore).
  • Idempotent Actuation: Dispatch control commands to target devices. Every command must be uniquely identified, versioned, and safely retryable.
  • Acknowledgement & Reconciliation: Track whether devices acknowledge and execute commands. Detect offline or unresponsive devices and raise alerts or fallback commands.
  • Audit & Replay: Maintain an immutable record of all raw telemetry, policy state transitions, and executed commands. Enable historical replay to validate new policy algorithms.

Non-Functional Requirements & Scale

  • Scale: 10,000,000 devices reporting telemetry every 5 seconds (~2,000,000 events/sec peak).
  • Actuation Latency: P99 end-to-end latency from telemetry anomaly to command issuance under 500 ms.
  • Out-of-Order / Network Partitions: Gracefully handle late-arriving (up to 10 minutes) and duplicate telemetry packets without issuing stale or duplicate commands.
  • Safety & Durability: Command audit trails must have zero data loss; actuation must fail safe if state is ambiguous.

What Your Design Should Cover

  1. Data Ingestion & Stream Processing: Stream partitioning strategy, event-time windowing, watermarks for late data, and deduplication.
  2. State Management & Hysteresis: How stateful aggregations are maintained and how hysteresis thresholds (e.g., trigger threshold vs. recovery threshold) prevent rapid flapping.
  3. Closed-Loop Actuation Protocol: End-to-end lifecycle of a control command (dispatch, lease/versioning, device ack, timeout handling, and dead-lettering).
  4. Data Modeling & Storage: Storage engines for high-volume time-series readings versus transactional command audit logs.
  5. Failure Modes & Disaster Recovery: Handling gateway disconnects, network split-brain, and validating system behavior via stream replay.

Hints

Hint 1

To prevent rapid control oscillation, decouple trigger thresholds from recovery thresholds using dual-threshold hysteresis states within a stateful stream processor.

Hint 2

Use monotonic sequence numbers or vector clocks per device along with command idempotency keys to ensure late telemetry or retried commands cannot trigger out-of-order state mutations.

Hint 3

Separate the high-throughput, loss-tolerant telemetry path (time-series stream) from the low-throughput, zero-loss actuation and audit path (distributed transactional ledger or log-compacted queue).

Follow-up questions

  • How do you handle a scenario where a device is disconnected for 15 minutes, reconnects, and dumps all buffered telemetry at once without triggering a cascade of stale commands?
  • If an entire partition of the stream processing cluster crashes during an active grid emergency, how do you prevent split-brain command dispatching upon recovery?
  • How would you implement an automated canary or shadow policy evaluation pipeline using the audit replay engine without sending live control signals?