Katkat
Back

Design Storage for a Flexible Metrics Monitoring System

AttentiveSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

This problem tests the ability to design a time-series storage system by balancing read/write throughput, data retention policies, and query performance at scale.

Problem Statement

Design a storage backend for a high-throughput metrics monitoring system. Your system must ingest time-series data (e.g., CPU utilization, request counts) from thousands of microservices and support performant queries for dashboards and alerting.

Functional Requirements

  • Ingestion: Support 100,000 writes per second of metrics with high availability.
  • Querying: Support time-range queries (e.g., "average CPU over the last 1 hour") and grouping by tags (e.g., host_id, service_name, region).
  • Rollups: Automatically aggregate data into coarser granularities (e.g., 1-minute samples into 1-hour averages) after 24 hours to save space.
  • Retention: Data should be stored at high resolution for 7 days, and at lower resolution for 30 days.

Constraints & Assumptions

  • Each metric point consists of a timestamp, a numeric value, and a set of key-value tags.
  • Cardinality of unique tag combinations can reach 1 million.
  • The system must handle late-arriving data (up to 5 minutes old).

Deliverables

  1. Data Model: How do you represent metrics and tags to allow efficient filtering?
  2. Storage Architecture: Would you use a relational database, NoSQL, or a specialized time-series database? Justify your choice.
  3. Partitioning & Indexing: Explain how you would partition the data to handle large volumes and optimize retrieval.
  4. Rollup Mechanism: Describe the process for downsampling old data.

Hints

Hint 1

Consider the trade-offs between a relational database's strict schema and a wide-column NoSQL store's flexibility for high-cardinality data.

Hint 2

Think about how to store tags: should they be indexed as individual columns, or stored in a compressed format within the record?

Hint 3

Consider utilizing a partitioned architecture based on time segments to facilitate efficient deletion of old data and to keep individual table/shard sizes manageable.

Follow-up questions

  • How would you handle a sudden spike in traffic that exceeds your system's write capacity?
  • How does your schema handle the addition of new tags dynamically without requiring expensive table migrations?
  • If your system experiences a partition, how do you handle data consistency versus availability (CAP theorem) for critical alerting metrics?