Katkat
Back

Reduce Cold-Food Incidents with Metrics and an Insulated-Bag Experiment

DoorDashData Scientist · Technical Screen
EasyData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

Design an A/B test and metric framework to evaluate whether equipping couriers with insulated bags reduces cold-food complaints while guarding marketplace efficiency.

Scenario

At DoorDash, customer support reports show that a noticeable portion of order refunds and negative ratings are driven by food arriving cold. The logistics team proposes distributing high-grade insulated delivery bags to couriers (Dashers) to address this issue.

You are the data scientist tasked with designing the evaluation strategy, establishing the metric hierarchy, and addressing key experimental trade-offs.

Questions to Answer

1. Metric Framework

  • Primary Metric: Define a clear, measurable primary metric to evaluate whether the initiative succeeds.
  • Secondary & Leading Indicators: What intermediate metrics reflect the causal chain between using the bag and the customer's perception of temperature?
  • Guardrail Metrics: What operational or marketplace metrics must not be degraded by requiring bag usage (e.g., Dasher drop-off, pickup duration)?

2. Experiment Design & Randomization Unit

  • Evaluate three potential units of assignment: Customer, Dasher, and Geographic Cluster (with Switchback).
  • Which unit do you recommend, and why?
  • How does marketplace interference (spillover) influence your choice?

3. Compliance and Treatment Effects

  • Not every Dasher who receives an insulated bag will remember or choose to use it on every delivery.
  • How would you measure compliance?
  • If you only measure the difference between couriers who received the bag versus those who did not, what type of estimate are you computing (e.g., Intention-to-Treat)? How would you estimate the effect of actually using the bag?

4. Diagnostic Case

Suppose you launch the test, and during the second week, cold-food reports suddenly increase in both treatment and control groups, with the treatment group showing a slightly larger increase.

  • What external and internal factors would you investigate first to explain this trend?

Hints

Hint 1

Consider whether a Dasher who receives a bag can be assigned orders alongside control Dashers in the same zone, and whether dispatch algorithms treat them differently.

Hint 2

Distinguish between Intention-to-Treat (ITT) and the Local Average Treatment Effect (LATE) using an instrumental variables framework when courier compliance is imperfect.

Hint 3

Look at macro factors like weather patterns or drop-off time inflation, as well as operational changes in restaurant prep times, when diagnosing sudden cross-variant metric spikes.

Follow-up questions

  • If Dashers spend an extra 45 seconds per delivery zipping and securing the bag, how does that compound across a high-density lunch rush?
  • How would you detect if customers in the treatment group are developing higher expectations because they see the Dasher arrive with a branded insulated bag?
  • If you decide on a switchback experiment across metro markets, how do you determine the duration of each time window to avoid carryover effects?