Katkat
Back

Design analysis to reduce cold-delivery complaints

DoorDashData Scientist · Technical Screen
HardData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This problem tests the ability to diagnose a service quality issue, define key performance indicators, and design an experiment to evaluate an operational intervention while controlling for confounding variables.

Scenario

DoorDash has observed a significant spike in customer complaints regarding 'cold food' deliveries. You have access to granular data including delivery timestamps, driver GPS coordinates, order preparation status, ambient temperature records, and customer feedback logs.

Task

Your goal is to define the problem and propose a roadmap to resolve it:

  1. Metric Definition: Define a primary metric to quantify 'cold-delivery' incidents. How would you validate that this metric correlates with actual customer satisfaction scores?
  2. Diagnostic Analysis: How would you isolate whether the bottleneck occurs during food preparation (at the restaurant) or during the transit phase (with the courier)? Describe the data cuts or segments you would analyze.
  3. Experimental Design: Suppose you propose a new feature: 'Thermal Bag Verification,' where couriers must upload a photo of their thermal bag before accepting an order. Design an experiment to measure the impact of this feature on cold food complaints.
    • How will you define the unit of randomization?
    • How will you handle potential network effects (spillover) where a treated courier impacts the experience of nearby un-treated couriers?
    • What would you use as your primary guardrail metrics to ensure this doesn't negatively impact the marketplace?

Constraints & Considerations

  • Assume a high volume of orders per city.
  • Consider how regional differences (weather, geography) affect your baseline expectations.
  • Focus on actionable insights rather than purely theoretical statistical models.

Hints

Hint 1

Consider normalizing delivery times against distance and traffic to create a 'delivery efficiency' score rather than relying on raw timestamps.

Hint 2

Think about using a clustered randomized design (randomizing by courier or by neighborhood) to minimize interference between control and treatment groups.

Hint 3

When evaluating the thermal bag feature, consider how the barrier to entry might affect courier supply and order acceptance rates as potential guardrail metrics.

Follow-up questions

  • If the feature shows a positive impact on complaints but a negative impact on order completion rates, how would you decide whether to roll it out?
  • How would you account for the 'Hawthorne Effect' where couriers might be more careful simply because they know they are being monitored during the experiment?
  • If you cannot perform a randomized controlled trial due to technical constraints, what quasi-experimental methods (e.g., Difference-in-Differences, Synthetic Control) would you use to measure the impact?