Katkat
Back

Design and Evaluate a Refund Policy for Delayed Orders

DoorDashData Scientist · Onsite
MediumData ScientistOnsiteAnalytics & Experimentation

Last updated

Design an automated, data-driven compensation policy for late deliveries on a three-sided marketplace. Formulate delay thresholds, balance customer lifetime value against operational costs, and structure an A/B test with appropriate randomization and guardrail metrics.

Scenario

On an on-demand food delivery platform, unexpected delays are a primary driver of customer churn, bad ratings, and inbound customer support tickets. Currently, customer refunds or credits for late deliveries are handled reactively when a customer contacts customer support.

Leadership wants to introduce a proactive or semi-automated refund/credit policy for delayed deliveries. Your goal as a Data Scientist is to design this policy end-to-end and evaluate it through an online experiment.

Tasks

1. Delay Definition & Eligibility Modeling

  • How would you define a delivery "delay" using historical platform data? (Consider promised estimated time of arrival (ETA), prep delays, courier transit time, and customer expectations).
  • How would you set severity tiers (e.g., minor, moderate, severe) to decide when compensation kicks in, and what historical analyses would you run to find the optimal cutoff points?

2. Compensation Structure & Trade-Offs

  • How do you determine the compensation type and amount (e.g., delivery fee refund, percentage discount, flat credit on the next order, full order refund)?
  • What framework would you use to balance platform cost against long-term customer retention (LTV) and restaurant/courier partner fairness?

3. Experimentation Design

  • Design an A/B test to evaluate the proposed automated delay compensation policy against the status quo (reactive support).
  • What is the unit of randomization (user, order, courier, market/geo cluster), and why? How do you account for network effects or spillover?
  • Define primary, secondary, and guardrail metrics across all three marketplace sides (consumers, merchants, couriers).
  • What is your sample size / duration determination strategy?

4. Abuse Prevention & Policy Robustness

  • Proactive refunds introduce potential moral hazard or opportunistic behavior (e.g., bad actors intentionally manipulating drop-off confirmations or address updates).
  • How would you detect, measure, and guard against fraud or gaming by consumers or couriers?

Hints

Hint 1

Analyze historical churn curves as a function of delay minutes past promised ETA to find the inflection point where churn risk jumps significantly.

Hint 2

Consider whether user-level randomization could lead to spillover or user dissatisfaction if repeat orders receive inconsistent policies across identical delays.

Hint 3

Differentiate between cash refunds back to the payment method versus platform credits; credits ensure customer reinvestment back into the marketplace and have a different economic cost.

Follow-up questions

  • If ETA prediction errors vary systematically by market density or weather, how would your delay policy account for external macro shocks without bankrupting the marketplace?
  • Suppose treatment shows a 1.5% lift in 30-day retention but increases total refund spend by 40%. How would you compute whether this policy is ROI-positive?
  • How would you ensure merchant partners are not unfairly blamed or penalized for delays caused primarily by courier dispatch bottlenecks?