Katkat

Design distributed median and mode

AnthropicSoftware Engineer · Onsite

Last updated

This question evaluates skills in distributed systems and large-scale analytics, covering data partitioning, streaming and sliding-window processing, quantile and heavy-hitter summarization, fault tolerance, and trade-off analysis for time, space, and network I/O.

You have a dataset with hundreds of billions of records spread across many machines. Design how to compute the global median and the global mode. Specify the data-partitioning strategy, local sketches or summaries, how reducers aggregate partial results, and how you minimize network I/O. Compare exact vs approximate approaches (e.g., quantile sketches, heavy-hitter sketches), discuss skew handling, fault tolerance, and incremental/streaming updates. Provide time, space, and communication complexity, and extend the design to a sliding-window median/mode over an unbounded stream.