Katkat
Back

Design a Metrics Storage Platform

OktaSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

This question assesses the ability to architect a scalable time-series database capable of handling high-velocity metric ingestion, efficient long-term storage, and sub-second query performance.

Problem Statement

Design a metrics storage platform capable of ingesting high volumes of telemetry data (counters, gauges, and histograms) from distributed services. Your system must support real-time monitoring and alerting, as well as long-term trend analysis.

Requirements

  • Functional: Support ingestion of tagged metrics, provide a query API for aggregation (e.g., sum, avg, p99), and support basic alerting triggers.
  • Non-functional: High write availability, scalable throughput (1 million writes/sec), and efficient disk usage for data older than 30 days.
  • Constraints: Assume a multi-tenant environment where performance isolation between tenants is required.

Deliverables

  1. API Design: Define the schema for submitting metrics and the query language interface.
  2. Data Model: Explain how to store tagged data (e.g., cpu_usage{host='web-01', env='prod'}) efficiently.
  3. Storage Strategy: Describe the database choice and how to handle data retention and downsampling.
  4. Architecture: Draw/describe the flow from metric ingestion to the query engine.

Example

  • Input: POST /v1/write with payload {"name": "http_requests", "tags": {"service": "auth"}, "value": 12, "ts": 1672531200}
  • Query: GET /v1/query?q=sum(http_requests{service='auth'})&start=1h&end=now

Considerations

  • How will you handle high-cardinality tags (e.g., user_id)?
  • How does the system handle bursty traffic?
  • How do you ensure consistency vs. availability during a partition?

Hints

Hint 1

Consider using an inverted index to map tag-value pairs to time-series identifiers for fast lookups.

Hint 2

Think about using a Log-Structured Merge (LSM) tree or a specialized time-series format like TSDB chunks to optimize for sequential writes.

Hint 3

Address the 'high cardinality' problem by discussing strategies like tag pre-aggregation or limiting allowed tag combinations per tenant.

Follow-up questions

  • How would you modify the design to support 'rolling up' metrics (e.g., turning raw data into 1-minute averages) after 24 hours to save space?
  • If one tenant experiences a massive spike in metrics, how would you prevent this from impacting the performance of other tenants?
  • What changes would be required to support global-scale querying across multiple geographic regions?