Katkat
Back

Evaluate a Bike Dasher Program with a Controlled Experiment

DoorDashData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

Tests marketplace experimentation design, tackling spatial-temporal spillover effects, metric selection, and decision criteria for a multimodal courier fleet.

Context

DoorDash is evaluating whether to formally scale a "Bike Dasher" program in dense metropolitan cores (e.g., Manhattan, Downtown San Francisco). Proponents argue that bicycle couriers reduce courier pickup dwell time by eliminating parking hassles and lowering fulfillment cost for short-distance orders. Skeptics worry about order basket size limits, slower long-distance transit times, weather sensitivity, and cannibalization of earnings for existing motorized Dashers.

You are the Lead Data Scientist tasked with designing the experimentation and measurement framework to determine whether to roll out this program.

Prompt

Walk through your end-to-end framework to evaluate this program:

1. Metric Architecture & Dwell-Time Measurement

  • How would you define and measure pickup dwell time using mobile telemetry and courier status transitions? What noise or measurement error would you anticipate?
  • Propose 1–2 primary success metrics, 2–3 guardrail metrics, and at least one ecosystem metric capturing supply-side health across both car and bike Dashers.

2. Experimental Design & Spillover Dynamics

  • Why does standard user-level or courier-level Bernoulli randomization violate the Stable Unit Treatment Value Assumption (SUTVA)?
  • Contrast a spatial cluster-based design (e.g., submarket / geohash clusters) with a time-based switchback design (e.g., alternating treatment and control windows in the same zone). Which do you recommend for dense downtown cores, and why?
  • How would you detect and measure spillovers—specifically, whether bike Dashers cherry-pick high-density, high-tip short orders and leave less profitable, high-mileage runs to car Dashers?

3. Launch Framework & Decision Rules

  • Assume your experiment reveals a 3-minute reduction in pickup dwell time and a 4% drop in cost-per-delivery for orders under 1.5 miles, but car courier utilization drops by 6% and merchant order cancellations for large orders increase by 1.5%.
  • Frame a concrete decision rule: how do you combine these trade-offs into an executive recommendation (Launch, Iterate, or Scrape)?

Hints

Hint 1

Consider how GPS drift and manual Dasher app status presses (e.g., 'Arrived at Store') can bias dwell-time calculations in high-rise downtown zones.

Hint 2

Bicycle and car couriers compete for the same real-time dispatch pool, meaning assigning couriers independently to treatment and control causes severe marketplace interference.

Hint 3

Switchback randomization controls for local market geography and supply density, but requires carryover washout windows between time blocks.

Follow-up questions

  • How would you account for severe weather shocks (e.g., sudden rainstorms) that disproportionately impact bicycle couriers during switchback windows?
  • If you observe order dispatch delays because the algorithm holds orders hoping a bike courier frees up, how would you quantify this dispatch-level latency cost?
  • How would you optimize the assignment dispatch logic so that car Dashers are not penalized with low-margin, long-tail deliveries when bike Dashers are active?