Interpret a Smart Wait Launch with Conflicting Metrics
WaymoData Scientist · Online Assessment
Last updated
Interpret a product launch where conservative wait estimates reduce conversion while observed pickup time also falls. Diagnose selection effects and alternative causes, define customer and fleet metrics with consistent denominators, and propose an experiment that can separate display accuracy from operational improvement.
A product changes its wait-time display to show more conservative estimates. After launch, conversion falls by 5% while actual time to pickup falls by 25%. A stakeholder concludes that fleet efficiency improved. Do you agree? Give alternative explanations for the pickup-time decrease and explain how the conversion change affects your interpretation.
Constraints & Assumptions
- The figures are observed before-after changes, not necessarily estimates from a randomized experiment.
- Conversion and time to pickup may be measured on different denominators.
- The display change can affect which users choose to proceed, even if fleet operations do not change.
Clarifying Questions to Ask
- How are conversion and time to pickup defined, and what are their denominators?
- Is time to pickup measured only for completed or accepted rides?
- Was the launch randomized or simultaneous with operational, geographic, or demand changes?
- Did rider mix, trip mix, supply, pricing, cancellations, or logging change?
- Is the 5% conversion drop relative or absolute, and what uncertainty surrounds both estimates?
Follow-up questions
- What would happen to observed pickup time if mostly long-wait users stopped converting?
- Which metric would more directly measure estimate accuracy?
- How would you design the randomization unit?
- What segment analysis would you pre-specify?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.