Katkat
Back

Diagnose and reduce cold-food refund costs

DoorDashData Scientist · Technical Screen
HardData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This problem tests the ability to diagnose root causes of operational cost drivers and design a rigorous experiment to validate potential mitigation policies.

Scenario

You are a Data Scientist at DoorDash. Our 'Cold-Food Refund' cost has increased by 15% month-over-month. You have access to granular order data, including order prep time, delivery time, courier temperature-controlled bag status, ambient weather conditions, and customer feedback logs.

Task

  1. Diagnostic Framework: Outline a methodology to isolate whether the increase is driven by merchant prep delays, courier transit inefficiencies, or external factors (e.g., weather or heavy traffic). Define the primary metrics you would monitor.
  2. Policy Intervention: Suppose you identify that orders sitting in the 'ready for pickup' stage for >10 minutes are the highest correlates with refund requests. Design an experiment to test a 'Dynamic Courier Dispatch' policy where couriers are dispatched closer to the predicted order completion time.
  3. Evaluation: How would you measure the causal impact of this policy? Discuss potential 'SUTVA' (Stable Unit Treatment Value Assumption) violations in this two-sided marketplace setting.

Data Assumptions

  • orders table: order_id, merchant_id, order_ready_time, courier_arrival_time, delivered_time, refund_amount, weather_conditions.
  • courier_logs table: courier_id, has_thermal_bag_flag, timestamp.

Example

If order A is ready at 12:00 PM but the courier arrives at 12:20 PM, the 'dwell time' is 20 minutes. If the refund rate for 20-minute dwell time is 4% compared to 0.5% for 2-minute dwell time, how do you adjust for other confounding factors like 'order size' or 'cuisine type'?

Hints

Hint 1

Use a regression-based approach or propensity score matching to control for confounding variables like cuisine type and order volume when calculating the impact of dwell time.

Hint 2

Consider the 'network effect' where dispatching a courier to one order affects the availability and waiting time for other pending orders in the same zone.

Hint 3

To measure causal impact, consider a randomized cluster design, such as geographic randomization, to minimize interference between treated and control groups.

Follow-up questions

  • How would you adjust your analysis if the refund data is 'missing not at random' (e.g., users who have previously been denied refunds are less likely to report cold food)?
  • If the 'Dynamic Courier Dispatch' policy reduces cold-food refunds but increases average 'Time-to-Door' for customers, how do you decide if the policy is a net positive?
  • Explain how you would use a 'Difference-in-Differences' (DiD) approach if you were forced to roll out this policy by entire city regions rather than individual orders.