Katkat

Design distributed word count without MapReduce

AdobeSoftware Engineer · Technical Screen

Last updated

This question evaluates a candidate's competency in designing scalable distributed systems and large-scale data processing pipelines, focusing on ingestion, partitioning/sharding, partial aggregation and merging, global top‑K computation, fault tolerance, idempotency, and storage/serving concerns.

Design a distributed system to compute word frequencies over terabytes of text without using MapReduce. Specify how you will ingest data (e.g., log or stream service), partition tokens across shards (e.g., consistent hashing with salting to mitigate hot keys), aggregate partial counts, and produce global results and top‑K across shards. Describe fault tolerance, idempotency, exactly‑once vs. at‑least‑once semantics, backpressure, recovery from node failures, and how results are stored and served.