Katkat
Back

Evaluate a Bike-Delivery Pilot with Marketplace Metrics

DoorDashData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This problem tests the ability to design an A/B test for a logistics marketplace, focusing on metric definition, experiment randomization strategy, and identifying potential confounding variables.

Scenario

DoorDash is launching a 'Bike-Only' delivery pilot in a dense urban market where parking-related delays for drivers are a significant source of latency. You are tasked with designing an experiment to measure the impact of this initiative on delivery times and marketplace health.

Your Task

  1. Metric Definition: Define one 'primary' metric to measure the success of the pilot and two 'guardrail' metrics to ensure the marketplace remains balanced.
  2. Randomization Strategy: How would you assign the experiment? Consider the trade-offs between assigning at the individual user level, the courier level, or the neighborhood (geographic) level.
  3. Interference & Spillover: Explain how 'network interference' (e.g., bike-couriers taking trips that car-couriers would have otherwise taken) might bias your results. How would you adjust your analysis or experiment design to account for this?

Contextual Constraints

  • The urban center is highly dense with limited parking.
  • Assume the pilot is limited to a specific neighborhood within the city.
  • Courier supply is split between those who own cars and those who own bikes.

Hints

Hint 1

Consider metrics that capture both the speed of delivery and the earnings stability of couriers who do not have access to bikes.

Hint 2

Geographic randomization is often preferred in logistics to prevent 'treatment leakage' where couriers move between zones, but it introduces the risk of population imbalance.

Hint 3

Think about the 'substitution effect'—if bike couriers increase, do car couriers see a decrease in total order volume, and does that change the marketplace's overall efficiency?

Follow-up questions

  • If the pilot shows a decrease in delivery times but a decrease in car-courier retention, how would you weigh these two conflicting outcomes?
  • How would you account for 'day-of-week' or 'weather' seasonality in your analysis of delivery times?
  • If you were to expand this to a second, less dense suburban market, would you expect the impact on delivery latency to be greater or smaller than in the initial urban pilot?