Katkat
Back

Design Biker metrics and A/B test plan

DoorDashData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests the ability to design an experimentation framework for a two-sided marketplace, focusing on metric definition, causal inference, and identifying potential gaming behaviors in delivery logistics.

Scenario

DoorDash is launching a pilot program to prioritize bicycle-courier assignments over cars in dense urban zones (e.g., Manhattan, San Francisco). You are asked to design an evaluation plan for this intervention.

Task

  1. Define one Primary Metric, two Secondary Metrics, and two Guardrail Metrics. Provide the exact mathematical formulas for each.
  2. Outline the experimental design for the A/B test. How will you randomize assignments? What is the appropriate unit of randomization?
  3. Describe three ways couriers or the algorithm might 'game' the primary metric, and propose a mitigation strategy for each.

Constraints

  • The primary metric must capture 'Efficiency'.
  • Assume you have access to order timestamps, courier transit mode, distance traveled, and customer tip data.
  • Consider the trade-offs between speed, cost, and courier earnings when choosing your metrics.

Hints

Hint 1

Consider that speed-to-customer is the primary efficiency goal, but ignoring the cost of the delivery could lead to unsustainable operations.

Hint 2

The unit of randomization must avoid interference between groups; consider whether randomizing by courier or by order is more appropriate for a two-sided marketplace.

Hint 3

Think about how couriers might manipulate order acceptance or how the system might prioritize short-distance, low-tip orders to inflate 'deliveries per hour' at the expense of earnings.

Follow-up questions

  • How would you adjust your power analysis if you suspect the 'SUTVA' (Stable Unit Treatment Value Assumption) is violated by bike-couriers stealing order volume from car-couriers?
  • If the pilot shows a significant increase in efficiency but a decrease in courier retention, how do you adjust your 'success' definition?
  • How would you account for weather variability or urban congestion cycles when calculating the confidence interval for your primary metric?