Design a Metrics Collection System

Scenario

Your organization operates thousands of services and batch processing pipelines across distributed clusters. To support real-time observability, alerting, and internal customer SLAs, you need to design a centralized metrics collection and query service.

The system must ingest metrics from applications and host daemons, aggregate time-series data, protect against cardinality explosions, store rollups efficiently, and serve queries for operational dashboards.

Requirements

Functional Requirements

• Metric Types: Support counters, gauges, and histograms/timers.
• Ingestion: Provide APIs or client SDKs for services to emit metrics with arbitrary key-value dimensions (labels).
• Cardinality Management: Prevent runaway dimension explosions (e.g., accidental inclusion of unique UUIDs or user IDs in labels).
• Query Interface: Support aggregation queries across dimensions (e.g., sum(rate(http_requests_total[5m])) by (service)), time-range windowing, and percentile computations (e.g., p95, p99 latency).
• Retention and Downsampling: Maintain raw granular metrics for short periods, and roll up older data for long-term trend analysis.
• Self-Monitoring: Alert when ingestion falls behind or when metric drops occur.

Non-Functional Requirements

• High Availability & Durability: Write path must handle bursts gracefully without dropping critical metrics; ingestion must remain available even if the query layer degrades.
• Low Latency Queries: Operational dashboards should load typical queries in under 500 ms.
• Scalability: Scale horizontally as host counts and service instances increase.

Scale Assumptions

• Throughput: 2,000,000 metric data points per second at peak.
• Active Series: Approximately 10,000,000 active unique time series at any given time.
• Retention:
  • Raw data (10-second resolution): 7 days.
  • 1-minute downsampled: 30 days.
  • 1-hour downsampled: 1 year.

Key Areas to Cover

1. Data Model & Ingestion Architecture: Push vs. pull model, client buffering, and gateway/queue design (e.g., Kafka/Pulsar).
2. Cardinality & Validation: How and where to detect and bound metric cardinality.
3. Storage Tiering: In-memory index, time-series storage engine (TSDB/LSM trees), and long-term object storage rollups.
4. Query Engine & Aggregations: How to serve real-time and historical queries without scanning massive unindexed datasets.
5. Fault Tolerance & Monitoring: Backpressure handling, dead-lettering, and self-health tracking.