Design a Count-Metrics Monitoring Platform

Scenario

Financial infrastructure platforms process millions of mission-critical operations per minute, such as payment authorizations, webhook deliveries, and API requests. Engineering and operations teams rely on a count-metrics platform to monitor traffic, compute rates (e.g., error rates, requests per second), slice counts across dimensional tags (e.g., merchant_id, http_status, currency), and fire alerts within seconds of an anomaly.

Design a distributed metrics ingestion, aggregation, and querying platform specialized for count metrics.

Requirements

Functional Requirements

• Metric Ingestion: Accept count increment events with timestamps, metric names, and key-value labels.
• Aggregation & Rates: Support real-time queries for rollups (e.g., 1-minute, 5-minute, 1-hour tumbling/sliding windows) and derived rates (e.g., count per second, error ratio).
• Dimensional Grouping: Allow filtering and grouping by arbitrary labels (e.g., group by merchant_id, status_code).
• Alerting Engine: Continuously evaluate user-configured alert rules (e.g., "alert if failed_charges / total_charges > 0.05 over a 5-minute window").
• Late Data & Deduplication: Handle late-arriving events (up to 1 hour late) and deduplicate idempotency keys or message IDs.
• Historical Replay & Retention: Allow replaying ingested logs to recompute metrics after logic changes or outages; maintain downsampled historical data.

Non-Functional Requirements

• Scale: Ingest 1,000,000 events/second at peak. Support query latencies under 500 ms for dashboards and under 30 seconds for alert evaluations.
• High Availability: 99.99% availability for ingestion; metric loss must be under 0.001%.
• Cardinality Protection: Defend against label cardinality explosion (e.g., accidental inclusion of unique request_id as a label).

Deliverables

1. High-Level Architecture: Ingestion pipeline, stream processing layer, storage engines, and alerting service.
2. Data Model & Aggregation Strategy: Time-bucketed counter formats, rollup hierarchies, and handling late/duplicate events.
3. Cardinality Control: Strategies for detecting, throttling, or dropping high-cardinality keys.
4. Storage & Retention Tiering: Separation of real-time and historical cold storage, and how metric replay works without impacting live traffic.