Katkat
Back

Design a Top Dasher experiment with interference

DoorDashData Scientist · Onsite
EasyData ScientistOnsiteAnalytics & Experimentation

Last updated

This problem tests your ability to design a controlled experiment in a marketplace with network effects, specifically focusing on how to mitigate spillover interference between treated and control groups.

The Problem

DoorDash wants to test a new "Top Dasher" incentive program in a specific city. The program offers priority access to high-value orders to couriers (Dashers) who maintain a 4.8+ rating and a 90% acceptance rate.

You are asked to design an experiment to measure the impact of this program on overall delivery speed and courier retention. However, because Dashers move between geographic zones and compete for the same orders, a standard user-level A/B test would result in severe interference: a treated Dasher taking an order prevents a control Dasher from taking it, artificially inflating the performance of the treated group.

Your Task

Design an experimental framework that addresses these points:

  1. Randomization Strategy: How will you assign treatment to avoid interference while maintaining statistical power?
  2. Metric Selection: Define the primary metric for the program and at least two guardrail metrics to monitor unintended negative impacts.
  3. Analysis: How would you account for the "spillover" effect if you were forced to run an experiment at the individual Dasher level?
  4. Scale: Briefly explain how you would calculate the minimum detectable effect (MDE) and sample size for your proposed design.

Hints

Hint 1

Consider using geographic clusters or time-based blocks (switchback) instead of individual user-level randomization.

Hint 2

Think about whether the interference happens more at the delivery-level (order competition) or the market-level (labor supply).

Hint 3

Guardrail metrics should focus on the experience of the control group (e.g., control group acceptance rates) and the customer experience (e.g., late deliveries).

Follow-up questions

  • How would your approach change if the primary goal shifted from 'courier retention' to 'reducing average delivery time'?
  • If you observe a significant positive impact in the treatment group, how do you verify that this is a true efficiency gain and not just a cannibalization of orders from the control group?
  • What are the specific trade-offs between a 'Cluster Randomized Trial' and a 'Switchback' design in terms of bias and variance?