Design distributed median and mode
AnthropicSoftware Engineer · OnsiteLast updated
This question evaluates skills in distributed systems and large-scale analytics, covering data partitioning, streaming and sliding-window processing, quantile and heavy-hitter summarization, fault tolerance, and trade-off analysis for time, space, and network I/O.
You have a dataset with hundreds of billions of records spread across many machines. Design how to compute the global median and the global mode. Specify the data-partitioning strategy, local sketches or summaries, how reducers aggregate partial results, and how you minimize network I/O. Compare exact vs approximate approaches (e.g., quantile sketches, heavy-hitter sketches), discuss skew handling, fault tolerance, and incremental/streaming updates. Provide time, space, and communication complexity, and extend the design to a sliding-window median/mode over an unbounded stream.
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.