Katkat
Back

How would you test a bike delivery option?

DoorDashData Scientist · Technical Screen
EasyData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This problem tests the ability to design an A/B test, define success metrics, and account for confounding variables when introducing a new logistics channel (bike delivery) to a marketplace.

Scenario

DoorDash is considering launching 'Bike Delivery' in a dense urban market where cars currently handle all orders. Bicycles offer the advantage of ignoring traffic jams and avoiding parking hassles but have limited range compared to cars. As a Data Scientist, you are tasked with designing an experiment to determine if launching bike delivery improves the overall marketplace health.

Your Task

  1. Metric Definition: Define one primary metric to measure success and two secondary guardrail metrics to monitor for unintended negative consequences.
  2. Experimental Design: Describe how you would set up an A/B test. Would you randomize by user, by restaurant, or by delivery zone? Justify your choice.
  3. Causal Inference: How will you distinguish the impact of bike delivery from general market fluctuations (e.g., seasonal changes or local events)?

Constraints

  • Assume you have access to historical order data, driver GPS logs, and delivery time metrics.
  • Your answer should focus on maintaining marketplace balance (ensuring both dashers and customers have a good experience).

Hints

Hint 1

Consider the difference between average delivery time and delivery time variance when selecting your primary metric.

Hint 2

Think about the 'network effect' or 'spillover effect' if you randomize by user in a small geographic area.

Hint 3

Think about how to handle the bias caused by the fact that bikes can only cover a limited radius compared to cars.

Follow-up questions

  • How would you account for the fact that bike delivery might increase the number of orders in high-traffic zones but decrease them in suburbs?
  • If the experiment shows a significant decrease in delivery times but a slight increase in order cancellations, how would you investigate the cause?
  • How would you determine if the bike delivery option is cannibalizing existing car-based delivery earnings rather than creating incremental value?