Design a Streaming Metrics Dashboard: Time-Range Queries and Event-Time Watermarks
Goldman SachsSoftware Engineer · Onsite
Last updated
Design the backend of a dashboard that shows metrics computed from a continuous event stream and lets users query any time range. The discussion focuses on answering range queries from pre-aggregated data and on how watermarks decide when event-time windows are complete despite late or out-of-order events.
Design the backend for a dashboard that shows metrics computed from a continuous stream of events and lets users look at any time range. The round was conversational rather than a structured whiteboard exercise, and two topics came up explicitly: how the system answers range queries over time, and how watermarks work.
Constraints and Clarifications
- Not given: what the dashboard displays, event volume, how fresh the numbers must be, how late events can arrive, retention, and how many people view the dashboard. Ask, or state your assumptions.
Clarifying Questions
- Which metrics does the dashboard show (counts, sums, rates, latency percentiles, distinct users), and broken down by which dimensions?
- How fresh must the newest data be, and may the last few minutes show provisional numbers that change later?
- Which time ranges and granularities do users ask for: the last hour at one-minute resolution, the last quarter at one-day resolution, arbitrary ranges?
- How late can events arrive, and must late events still be counted?
- What is the event volume, and how long must raw events and aggregates be kept?
Hints
Hint 1
Do the work before the query: Decide what can be aggregated ahead of time so that a query over a long range never has to touch raw events, and which kinds of metrics can still be combined after they have been aggregated.
Hint 2
Event time is not arrival time: An event can reach the system well after it happened. Decide when the numbers for a given minute are final, and what happens when a straggler for that minute shows up afterwards.
Follow-up questions
- One input partition stops receiving events. What happens to the watermark and to the dashboard, and how do you prevent it?
- Users want the p99 latency over the last 30 days. What do you store so that it can be computed without the raw events?
- Yesterday's numbers changed after a batch of late events arrived. How do you limit and communicate such corrections?
- One dimension value produces most of the events. How do you keep its aggregation from becoming a bottleneck?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.