Katkat
Back

Design a Distributed Metrics Counter

StripeSoftware Engineer · Onsite
HardSoftware EngineerOnsiteSystem Design

Last updated

This question tests the ability to architect a highly available, scalable system for ingesting and aggregating high-throughput telemetry data while managing trade-offs between consistency and performance.

Problem Statement

Design a distributed metrics platform that can ingest and aggregate events at a scale of 10 million events per second across multiple global regions. The system must support two primary access patterns: real-time dashboarding (last 5 minutes) and long-term historical analysis (last 30 days).

Functional Requirements

  • Ingestion: Accept incoming counter increments via an HTTP API (e.g., increment(metric_id, value, timestamp)).
  • Aggregation: Support rolling up metrics by various granularities (e.g., 1-second, 1-minute, 1-hour).
  • Querying: Retrieve current values and historical trends for a given metric_id.

Non-Functional Requirements

  • Latency: Ingestion must have low latency (sub-50ms) as it sits in the critical path of client applications.
  • Scalability: The system must handle sudden traffic spikes (e.g., 10x traffic bursts).
  • Consistency: The system should prioritize availability and performance over strict strong consistency (eventual consistency is acceptable).

Deliverables

  1. High-level architecture diagram (described in text).
  2. Choice of storage layer (e.g., Time Series Database vs. NoSQL) and why.
  3. Strategy for handling high-volume write contention for individual metrics.
  4. Design for the rollup/aggregation pipeline.

Hints

Hint 1

Consider how to decouple ingestion from storage using a message queue or buffer to absorb traffic spikes.

Hint 2

Think about using a two-tier storage approach: an in-memory cache for hot/recent data and a distributed TSDB for cold/archived data.

Hint 3

Evaluate the trade-off between using a centralized aggregator versus local, per-node aggregation before flushing to storage.

Follow-up questions

  • How would you handle the 'hot key' problem if a single metric (like 'total_requests') receives an extreme volume of increments?
  • How would you implement data retention policies to automatically expire or downsample data older than 30 days?
  • If the system requires 99.99% accuracy, how does that change your approach to distributed aggregation compared to allowing approximate results (like HyperLogLog)?