Katkat
Back

Design a Scalable Metrics Monitoring System

AmazonSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Design a high-throughput time-series metrics monitoring and alerting platform similar to Amazon CloudWatch, focusing on series identity, cardinality controls, and low-latency aggregations.

Scenario

Design an end-to-end metrics monitoring platform (similar to Amazon CloudWatch or Prometheus) that allows thousands of microservices and infrastructure components to emit metrics, visualize them via real-time dashboards, run ad hoc analytical queries, and evaluate alerting rules.

Functional Requirements

  • Metric Ingestion: Accept metric data points containing a metric name, timestamp, numeric value, and arbitrary key-value dimensions/tags.
  • Dashboards & Queries: Support aggregations (such as sum, avg, p90, p99) across specified time windows (e.g., 1-minute, 5-minute buckets) over arbitrary tag filters.
  • Alerting: Continuously evaluate threshold-based alerting rules (e.g., error_rate > 5% for 3 consecutive minutes) and dispatch notifications reliably.
  • Data Retention & Compaction: Retain high-resolution data for short-term debugging and downsampled data for long-term trend analysis.

Non-Functional Requirements & Scale

  • Ingestion Scale: 10 million data points per second sustained write throughput, with peaks up to 25 million points per second.
  • Data Size: Average raw sample size of 100 bytes (~1–2.5 GB/s ingestion bandwidth).
  • Query Latency: Sub-second response times (< 500 ms) for standard dashboard queries covering the last 24 hours.
  • Alert Latency: Alerts should trigger within 1 minute of a breach condition.
  • High Availability & Durability: 99.99% availability for ingestion; zero silent data loss for recorded metrics.

Key Areas to Cover

  1. Data Model & Series Identity: How a time-series is uniquely identified, indexed, and compressed in memory and on disk.
  2. Architecture: End-to-end data flow spanning edge collection agents, ingestion buffers, time-series storage engines, query execution, and the alerting pipeline.
  3. Cardinality Management: Strategies to detect, mitigate, and throttle cardinality explosions caused by high-entropy tags.
  4. Downsampling & Retention: Storage tiering (hot vs. cold) and background compaction/rollup strategies.
  5. Alert Evaluation: Idempotent, retry-safe alert evaluations that handle delayed or out-of-order data points.

Hints

Hint 1

Differentiate between the series index (mapping metric names and tag sets to an internal series ID) and the sequential data point storage (timestamp-value pairs).

Hint 2

Leverage specialized time-series compression algorithms like Gorilla/XOR for floating-point values and delta-of-delta encoding for timestamps.

Hint 3

Decouple real-time alert evaluation from disk-bound ad hoc queries by maintaining a short-term sliding window buffer in memory or streaming workers.

Follow-up questions

  • How does your system handle late-arriving or out-of-order data points without corrupting precomputed rollups?
  • How would you design rate limits and safeguards to prevent an engineer from accidentally pushing UUIDs as tag values and exhausting system memory?
  • If an alert evaluation worker crashes mid-evaluation, how do you ensure alerts are neither dropped nor duplicated across failover nodes?