Design a Fleet Metrics Monitoring Platform, Then Go Deep

Design a platform that monitors a fleet of services and hosts. Each instance emits numeric measurements over time — request counts, error counts, latencies, CPU and memory usage — and the platform needs to collect them, store them as time series, let engineers build dashboards against them, and alert an on-call engineer when a rule is violated.

You're not expected to design every box in equal depth. Sketch the full pipeline briefly first, then pick one or two areas and go deep.

Clarifying Questions

• Roughly how many hosts and services are being monitored, how many distinct measurements does each emit, and at what interval? How many distinct active time series does that imply?
• Does each instance push its own measurements out, or does something poll each one on a schedule?
• For how many months does history need to stick around, and does it stay at full resolution the whole time?
• Can a dashboard tolerate showing slightly stale numbers, and can an alert tolerate occasionally missing a handful of points without it mattering?
• Is this platform only for numeric measurements, or does it also need to ingest logs or distributed traces?
• Once a rule fires, how does the notification reach a human, and do related alerts need to be grouped or temporarily muted?

Signals of a Strong Design

• A precise shape for a single measurement and the identifying labels attached to it, with numbers derived from that shape for how fast points arrive, how many series stay active, and how much space they occupy
• A short end-to-end pipeline sketch, followed by a clearly justified choice of where to go deep
• Storage suited to how time-series data is actually written (append-heavy, time-ordered) and read (recent data at full resolution, old data mostly over wide ranges)
• For a storage deep-dive: compression, partitioning by time, and a downsampling or rollup strategy as data ages
• For a cardinality deep-dive: what causes a label combination explosion, and how to bound or detect it before it degrades the whole system
• For an alerting deep-dive: how evaluation stays correct and timely even as the number of active rules and series grows, and how a single noisy series is kept from paging someone needlessly

Scale to reason from

Assume on the order of 50,000 hosts, each running a handful of services and emitting on the order of a few hundred distinct metric names at a 10-second interval, with labels for host, service, and deployment version. That alone implies tens of millions of active series once every label combination is counted — the cardinality estimate is usually the first thing worth writing down, because it decides which storage and query approaches are even feasible.