Katkat
Back

Design a Metrics Monitoring Dashboard

Walmart LabsSoftware Engineer · Technical Screen
MediumSoftware EngineerTechnical ScreenSystem Design

Last updated

This question tests the ability to architect a scalable time-series data pipeline, focusing on ingestion, storage strategies for high-throughput metric data, and query performance for dashboard visualization.

Problem Statement

Design a metrics monitoring platform capable of collecting, storing, and visualizing time-series data (e.g., CPU utilization, request latency, error rates) from thousands of microservices.

Functional Requirements

  1. Ingestion: Accept incoming metrics via a push-based API (e.g., POST /v1/metrics).
  2. Storage: Store metrics with a retention period of 30 days.
  3. Querying: Support a dashboard API to retrieve aggregate data over specific time ranges (e.g., "average latency for service X over the last hour").

Non-Functional Requirements

  1. Scale: Handle 1 million metrics per second.
  2. Low Latency: Queries should return results in under 500ms for standard time ranges.
  3. Availability: Ensure the system is highly available, as it is critical for infrastructure observability.

Your Deliverables

  1. Data Model: Define how you structure the metric points (e.g., metric name, timestamp, value, labels).
  2. Architecture: Draw the high-level components (API gateways, ingestion workers, storage layer, query engine).
  3. Storage Strategy: Explain your choice of database (e.g., TSDB, RDBMS, NoSQL) and how you handle downsampling or data compression over time.
  4. Trade-offs: Discuss the trade-offs between consistency and availability for this write-heavy system.

Hints

Hint 1

Consider how to efficiently handle high-cardinality labels (e.g., different server IDs or container IDs) associated with a single metric.

Hint 2

Think about using a message queue between the ingestion API and the storage worker to buffer spikes in traffic.

Hint 3

Discuss how downsampling (e.g., aggregating 1-second data points into 1-minute averages) helps improve dashboard query speed for longer time ranges.

Follow-up questions

  • How would you implement alerting based on these metrics? What components would you add to detect and notify on threshold breaches?
  • If the system experiences a network partition, should the ingestion layer favor consistency or availability? Why?
  • How would you secure the ingestion endpoint to prevent unauthorized services from pushing malicious or noisy metric data?