Design a Large-Scale Real-Time Temperature Collection System

Scenario

Large retail operations rely on continuous cold-chain monitoring across thousands of grocery stores, fulfillment centers, and refrigerated delivery trucks. Perishable goods spoil quickly if temperatures deviate from required thresholds. You need to design a distributed, real-time temperature ingestion and monitoring platform that securely collects, processes, and persists sensor telemetry at scale.

Scale & Traffic Assumptions

• Sensors: 5,000,000 active temperature sensors deployed globally.
• Reporting Interval: Each sensor transmits a reading every 10 seconds.
• Ingestion Throughput: ~500,000 events/second average, with bursts up to 1,000,000 events/second (e.g., reconnected transit fleets).
• Retention: Raw telemetry preserved for 30 days; downsampled hourly aggregates preserved for 2 years.
• Latency: Real-time current status query latency under 50 ms at p99; ingestion-to-alert latency under 5 seconds.

Requirements

Functional

1. Secure Ingestion: Securely ingest readings via lightweight protocols (e.g., MQTT, HTTPS/gRPC over TLS with device authentication).
2. Time Handling & Idempotency: Support both event_timestamp (device clock) and arrival_timestamp (server clock); handle duplicate emissions and out-of-order packets idempotently.
3. Current State Projection: Provide low-latency lookups for the latest known temperature and health status of any sensor, freezer unit, or facility.
4. Historical Storage & Downsampling: Persist time-series readings with automated rollups (e.g., min, max, avg per hour/day) and handle late-arriving backfilled data.
5. Alerting Support: Support rules-based triggering when readings deviate outside safe bands for sustained durations.

Non-Functional

• High Availability & Fault Tolerance: 99.99% write availability. Disconnected facilities or transit vehicles must be able to buffer locally and upload in batches upon reconnection without overwhelming the system.
• Scalability: Horizontal scaling of ingestion gateways, processing workers, and storage partitions.

What to Cover

• Architecture: High-level topology including edge proxies/gateways, message buffering, streaming consumers, and data stores.
• API & Data Model: Telemetry packet payload, ingestion API, and data storage models (current-state cache vs. historical time-series storage).
• Late Data & Out-of-Order Handling: Strategies for watermark management, updating historical aggregations when delayed data arrives, and avoiding stale overwrites in the current-state store.
• Data Lifecycle & Downsampling: Mechanism for downsampling raw data into long-term rollups and pruning raw logs.