Katkat
Back

Design Global Metrics Monitoring and Real-Time Alerting

ZscalerSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

This question tests the ability to design a distributed, scalable telemetry pipeline capable of handling high-cardinality time-series data and real-time alerting with deduplication.

System Design: Global Metrics and Alerting

Design a global monitoring and alerting system for a microservices architecture spanning 10+ regions. The system must collect metrics from thousands of service instances, evaluate alerting rules against these metrics, and notify on-call engineers.

Requirements

  1. Ingestion: Handle ingestion from globally distributed services with varying network latencies.
  2. Storage: Design a time-series storage strategy that supports efficient querying for dashboards and historical evaluation.
  3. Alerting: Support stateful alerting rules (e.g., 'trigger if error rate > 5% for 3 minutes').
  4. Notification: Implement a mechanism to deduplicate alerts so that engineers aren't overwhelmed by alert storms during a regional outage.
  5. Cardinality: Propose a strategy to manage high-cardinality metric labels (e.g., user_id, request_id) to prevent storage explosion.

Scale Assumptions

  • 500,000 unique metric series per region.
  • 5 million data points ingested per second globally.
  • Alert notification latency should be under 30 seconds.

Expected Output

  • Define the architecture components (Collector, Aggregator, Storage, Alert Engine).
  • Explain the data flow from a service instance to a dashboard/notification.
  • Address how to handle late-arriving data and regional failures.
  • Discuss the trade-offs between centralized vs. decentralized storage.

Hints

Hint 1

Consider a hierarchical aggregation model where regional collectors perform local rollups before sending data to a global store.

Hint 2

For high cardinality, discuss techniques like dropping labels at the source, using bloom filters, or implementing a specialized metric index.

Hint 3

To solve for alert storms, use a grouping mechanism based on alert definitions and timestamps to coalesce multiple alerts into a single notification.

Follow-up questions

  • How would you ensure the monitoring system itself remains functional if the services it monitors are undergoing a massive outage?
  • If your storage engine uses a fixed retention policy, how would you implement 'downsampling' to retain long-term trends without losing critical resolution?
  • How would you verify the correctness of the alerting engine when an alert rule is updated or deleted?