Katkat
Back

Design a metrics platform without alerting

LinkedInSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteSystem Design

Last updated

This problem tests your ability to design a high-throughput time-series metrics system, focusing on ingestion pipelines, efficient storage schema for multi-dimensional data, and query performance.

Problem Statement

Design a scalable time-series metrics platform that supports the collection, aggregation, and querying of metrics (e.g., CPU usage, request counts, latency histograms). Assume the system needs to handle millions of incoming data points per second with high availability.

Functional Requirements

  • Ingest metrics consisting of a name, timestamp, value, and a set of key-value pairs (labels).
  • Support three primary types: Counters (monotonically increasing), Gauges (instantaneous values), and Histograms (distribution of values over time).
  • Provide a query interface to retrieve metric trends over specific time ranges and group/filter by labels.
  • The system does not require an alerting component.

Non-Functional Requirements

  • High write throughput and low query latency.
  • Data retention for at least 30 days.
  • Horizontal scalability and support for multi-tenancy.

What to include in your design:

  • API design for metric ingestion and query execution.
  • Data model choice and database selection (e.g., TSDB, columnar, key-value).
  • Architectural components including ingestion pipeline, storage layer, and query engine.
  • Strategies for handling high cardinality in labels (e.g., metric names combined with unique user IDs).

Hints

Hint 1

Consider how you would structure the data to handle frequent label-based filtering, perhaps using an inverted index.

Hint 2

Think about how to aggregate raw metric points as they age to save storage space and improve query performance over long time windows.

Hint 3

Discuss the trade-offs between a push-based model (clients send metrics) versus a pull-based model (server scrapes metrics).

Follow-up questions

  • How would you handle a sudden surge in traffic to ensure the ingestion pipeline does not crash?
  • If a single metric name has thousands of unique label combinations, how does that impact your storage design and what steps can you take to mitigate performance degradation?
  • How would you implement data downsampling/compaction to support queries covering multi-month time ranges?