Katkat
Back

Design a Metrics System

SnapchatSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Design a high-throughput, low-latency metrics collection, aggregation, and querying platform. The question tests your ability to handle multi-dimensional time-series data, mitigate high-cardinality label explosions, and design tiered storage for varying query windows.

Scenario

At a large-scale multimedia messaging platform, internal infrastructure and customer-facing microservices emit telemetry data continuously. You are tasked with designing a unified metrics collection and monitoring platform. The system must collect, store, and serve time-series metrics used for real-time operational alerting, service dashboards, and capacity planning.

Requirements

Functional Requirements

  • Metric Ingestion: Support standard metric types: counters (monotonically increasing), gauges (instantaneous values), and histograms/timers (distribution percentiles like p50, p95, p99).
  • Data Model: Support multi-dimensional metric naming with dynamic key-value tags (e.g., http_requests_total{service="story_service", region="us-east-1", status="500"}).
  • Querying & Alerting: Provide an expressive query interface to aggregate metrics across dimensions (e.g., sum, rate, quantile) over sliding windows (e.g., 1m, 5m, 1h) with sub-second alert evaluation.
  • Data Retention & Downsampling: Support short-term high-resolution data (e.g., 10-second granularity for 7 days) and downsampled long-term retention (e.g., 5-minute granularity for 90 days, 1-hour granularity for 1 year).

Non-Functional Requirements

  • Scale: 10 million distinct time series ingested per second, totaling ~50 billion metric data points per day.
  • Query Latency: Dashboard queries over the last 1–6 hours must return in under 500 ms; ad-hoc queries over 30 days should return in under 3 seconds.
  • Cardinality Defense: Prevent runaway tag explosions (e.g., someone accidentally tagging user IDs or UUIDs) from degrading system availability.
  • Fault Tolerance: The system must degrade gracefully during ingestion spikes without dropping critical operational telemetry.

What Your Design Should Cover

  1. Data Model & Ingestion Pipeline: Agent/client library design (push vs. pull), wire protocols, client-side pre-aggregation, and stream buffer architecture (e.g., Kafka).
  2. Storage Architecture: Choice of Time Series Database (TSDB) or LSM-tree/columnar store, index structure for time-series IDs (series keys to postings lists), chunking, and compression (e.g., Gorilla / double delta encoding).
  3. Cardinality Management: Mechanisms for detecting, rate-limiting, or dropping high-cardinality series before they exhaust index memory.
  4. Downsampling & Compaction: Asynchronous roll-up pipelines and retention policies.
  5. Failure Modes & Observability: How to handle partition failures, backpressure, and how the metrics system monitors its own health.

Hints

Hint 1

Differentiate the index path (mapping dimension tags to a unique Series ID) from the data path (storing timestamp and float value pairs).

Hint 2

Consider client-side or collector-level aggregation for histograms and counters to drastically cut the network payload and write TPS.

Hint 3

Evaluate Gorilla-style compression (XOR delta-of-deltas for timestamps and XOR for floating-point values) to optimize memory and disk footprint.

Follow-up questions

  • How would you implement precise percentile calculation across multiple distributed nodes without transmitting raw data points?
  • If an engineer commits a bug that attaches a dynamic UUID to every emitted metric tag, how does your ingestion pipeline detect and quarantine this leak before the inverted index OOMs?
  • How does your query engine optimize federated queries that span both in-memory real-time buffers and cold, long-term object storage?