Katkat
Back

Measure Impact of Merchant Variety on Consumer Experience

DoorDashData Scientist · Onsite
MediumData ScientistOnsiteAnalytics & Experimentation

Last updated

This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Measure Impact of Merchant Variety on Consumer Experience states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Scenario

DoorDash's product team is exploring how merchant variety/selection affects consumer experience and marketplace health, and is considering expanding the breadth or evenness of available merchants.

Question
  1. How would you define merchant variety or selection in an operational, measurable way?
  2. What success metrics would you track at both the merchant and consumer levels, plus marketplace guardrails?
  3. How would you design an A/B test (or an alternative quasi-experimental design) to evaluate the effect of a change in variety, and how would you analyze the results?
Hints
  • Define variety along breadth, depth, evenness, coverage, and novelty; use diversity indices (Shannon entropy, effective number of categories, Simpson diversity / HHI).
  • Cover conversion, retention, order frequency, revenue per user, plus reliability and unit-economics guardrails.
  • Choose the unit of randomization by the lever (user/session for a ranking change, geo-cluster for a supply change); discuss interference, power (ICC/design effect, CUPED), heterogeneous effects, and cannibalization.