Katkat
Back

Evaluate Courier-Selected Delivery Distance Limits

DoorDashData Scientist · Onsite
MediumData ScientistOnsiteAnalytics & Experimentation

Last updated

This question tests your ability to design an A/B test for a marketplace feature while accounting for network effects, trade-offs between supply-side control and demand-side fulfillment, and selecting appropriate metrics.

The Scenario

DoorDash is considering a new 'Delivery Radius Preference' feature, allowing couriers to set a maximum distance for the offers they receive. Couriers argue this increases job satisfaction and reduces vehicle wear, while operations managers worry it will lead to 'delivery deserts' for long-distance orders, increasing wait times and customer churn.

Your Task

You are tasked with designing a pilot study to evaluate the impact of this feature. Address the following components:

  1. Experimental Design: Given the marketplace nature of the platform, what is your unit of randomization (e.g., courier, city, time-block) and why? How will you handle spillover effects (e.g., a courier in the treatment group 'stealing' short-distance orders from a courier in the control group)?

  2. Metric Framework: Define your key success metrics. How do you propose to balance courier-centric metrics (earnings, retention) with customer-centric metrics (delivery time, order cancellation rate)?

  3. Evaluation Strategy: How would you determine if the feature is 'net-positive' for the marketplace if courier earnings increase but customer wait times also increase?

Constraints

  • Assume the platform operates in thousands of distinct markets with varying population densities.
  • You have access to granular data on every order (offer time, courier location, order distance, tip amount, delivery success).

Hints

Hint 1

Consider geographic clustering, such as a switchback design or city-level randomization, to mitigate network interference.

Hint 2

Think about the difference between primary metrics (e.g., net order volume) and guardrail metrics (e.g., average delivery time for long-distance orders).

Hint 3

Account for the 'opportunity cost' of the courier; if couriers reject more orders, who picks them up, and at what cost to the platform?

Follow-up questions

  • How would you adjust your experimental design if you discovered that long-distance deliveries were critical to keeping couriers active during off-peak hours?
  • If the feature leads to a significant increase in courier retention but a 5% decrease in on-time delivery for long-distance orders, what business logic would you use to decide whether to roll out the feature?
  • How could you use historical data to simulate the likely impact on order acceptance rates before running a live experiment?