Katkat
Back

Design a Low-Latency Metrics and Alerting Platform

CitadelSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

Design a high-throughput, multi-tenant metrics ingestion and alerting engine capable of triggering critical alerts within two seconds. The question evaluates your knowledge of streaming architectures, time-series storage, cardinality management, and fault-tolerant notification delivery.

Scenario

Trading environments and mission-critical financial infrastructure generate massive streams of time-series telemetry. You are tasked with designing a multi-tenant metrics collection, storage, and alerting platform. The system must ingest raw metrics from thousands of distributed hosts and services, serve dashboard queries, and evaluate alerting rules with sub-two-second detection for critical failures.

Requirements

Functional Requirements

  • Metric Ingestion: Support multi-tenant ingestion of structured metrics (metric_name, key-value tags, timestamp, value).
  • Dashboards & Queries: Support ad-hoc range queries and aggregate rollups (e.g., rate, percentile, sum, average) with sub-second dashboard response times for recent data.
  • Low-Latency Alerting: Evaluate threshold alerts and service level objective (SLO) breaches, firing critical notifications to downstream sinks (PagerDuty, Webhooks, internal buses) within 2 seconds of the event timestamp.
  • Liveness & Dead-Man's Snitches: Detect missing heartbeats (e.g., silent feed drops or process crashes) reliably.
  • Alert Lifecycle & Deduplication: Manage alert states (PENDING, FIRING, RESOLVED), suppressing flapping and deduplicating alert deliveries across redundant pipelines.

Non-Functional Requirements

  • Latency: End-to-end alert detection and dispatch latency under 2 seconds (P99).
  • High Availability: 99.999% availability for critical alerting; resilience to regional network partitions.
  • Cardinality Protection: Dynamic rate limiting and quota enforcement to prevent runaway metrics from degrading cluster stability.
  • Durability & Retention: Tiered retention (e.g., raw data for 14 days, 1-minute rollups for 90 days, 1-hour rollups for 1 year).

Scale & Assumptions

  • Throughput: 20 million metric samples ingested per second globally.
  • Volume: ~2 KB per metric packet including metadata; ~40 GB/sec uncompressed network throughput.
  • Active Rules: 50,000 active alerting rules, with 5,000 classified as high-priority real-time alerts.
  • Tenancy: Hundreds of independent teams and services sharing the platform.

Key Discussion Areas

  • Architecture: Push vs. pull ingestion, streaming alert evaluation engine vs. periodic TSDB query evaluation.
  • High Cardinality: Defense mechanisms against label explosions at ingestion gateways.
  • Late Data & Out-of-Order Samples: Handling delayed telemetry without producing false positives or flapping alerts.
  • Reliability: Active-active regional setups and masterless alert deduplication.

Hints

Hint 1

To achieve sub-two-second alerting at scale, separate the alerting pipeline into an in-memory streaming path instead of relying on periodic database query polling.

Hint 2

Mitigate label cardinality explosions at the ingestion edge using per-tenant bloom filters or token-bucket rate limiters mapped to unique label sets.

Hint 3

Use a shared distributed lock or an active-active gossip/consensus layer (such as an Alertmanager mesh) to deduplicate notifications across redundant evaluation workers.

Follow-up questions

  • How does your alerting pipeline handle out-of-order or backfilled metrics without triggering false alert resolutions or duplicate firing events?
  • If a network partition isolates a data center, how do you prevent alert storms while ensuring dead-man switch alerts still fire correctly?
  • How would you compress and store time-series data on disk to balance write throughput with rapid aggregate queries (e.g., Gorilla / double-delta encoding)?