Katkat
Back

Design a Real-Time Monitoring System

DoorDashSoftware Engineer · Onsite
EasySoftware EngineerOnsiteSystem Design

Last updated

This problem tests your ability to design a high-throughput observability pipeline capable of ingesting, storing, and alerting on time-series metrics from a distributed fleet of services.

Scenario

Design a real-time monitoring system for a microservices architecture similar to DoorDash. The system must collect metrics (e.g., latency, error rates, CPU usage) from thousands of service instances and provide dashboards and alerting capabilities.

Requirements

  • Scale: The system must ingest 1 million data points per second. Data points consist of (timestamp, metric_name, service_id, value).
  • Retention: Metrics must be queryable at 1-second resolution for 7 days, and at 1-minute resolution for 1 year.
  • Alerting: The system should trigger alerts when a metric exceeds a specific threshold (e.g., error rate > 5% over a 1-minute window).
  • Availability: The monitoring system should have high availability and be fault-tolerant to node failures.

Expectations

Your design should cover:

  1. High-level architecture: Data flow from service instances to storage.
  2. Storage strategy: Schema design, choice of database (SQL vs. NoSQL vs. Time-Series), and how you handle data downsampling.
  3. Alerting mechanism: How the system evaluates conditions efficiently across a distributed dataset.
  4. Scalability and Trade-offs: How to handle traffic spikes and the trade-offs between consistency and availability in your design.

Hints

Hint 1

Consider using a push-based model (agents pushing to a collector) versus a pull-based model (server scraping metrics).

Hint 2

Time-series data is often immutable and append-only; consider using specialized databases like Prometheus, InfluxDB, or Cassandra which handle these patterns effectively.

Hint 3

For alerting, look into a stream processing architecture to compute rolling windows instead of repeatedly querying a database.

Follow-up questions

  • How would you handle high cardinality metrics (e.g., metrics tagged by user_id) which could cause memory issues in storage?
  • If the monitoring system itself starts to fail due to a traffic spike, what strategies would you use to prevent cascading failures in the monitored services?
  • How would you modify your design to support tracing (e.g., OpenTelemetry spans) in addition to basic metrics?