Katkat
Back

Design and analyze batching algorithm experiment

DoorDashData Scientist · Onsite
HardData ScientistOnsiteAnalytics & Experimentation

Last updated

This problem tests the ability to design a cluster-randomized experiment in a marketplace setting, focusing on managing network effects (spillover) and defining appropriate inference strategies for geo-based batches.

The Scenario

DoorDash is testing a new 'Dynamic Batching' algorithm intended to group more orders together to increase fleet efficiency. Because delivery drivers operate across a shared geographical area, assigning a driver in the 'Treatment' group to a batch might pull them away from orders in the 'Control' group, creating a negative spillover effect.

Your Task

Design an experiment to evaluate this algorithm. Your proposal should address the following:

  1. Randomization Strategy: Describe how you would define your randomization units. If you choose a geo-based cluster approach, explain how you would handle 'border' effects between treated and control clusters.
  2. Metric Specification: Define a primary metric to measure efficiency. List at least two guardrail metrics to ensure the algorithm doesn't negatively impact the consumer or restaurant experience.
  3. Causal Inference: How would you account for the cluster-level interference (spillover) when calculating the treatment effect? Provide the conceptual approach for calculating the variance of your estimator.
  4. Power Analysis: Given that you are randomizing by geo-cluster rather than by order, explain how the effective sample size is impacted and what adjustments must be made to your power calculation.

Assumptions

  • You have 500 distinct delivery zones (geo-clusters) available.
  • You have a pre-experiment period of 4 weeks of historical data.
  • Assume the goal is to increase 'Orders per Driver-Hour' without decreasing 'Average Delivery Time' by more than 2%.

Hints

Hint 1

Consider implementing a 'buffer zone' of excluded areas between treated and control clusters to minimize the impact of cross-cluster driver movement.

Hint 2

When calculating standard errors, use cluster-robust standard errors to account for the fact that observations within the same zone are correlated.

Hint 3

Intraclass Correlation Coefficient (ICC) is critical here; use historical variance within zones versus between zones to adjust your power expectations.

Follow-up questions

  • How would you adjust your decision framework if the experiment shows a significant lift in efficiency but a borderline violation of your guardrail metric?
  • If the algorithm works better in dense urban environments than in suburban ones, how would you design a follow-up analysis to validate this heterogeneity?
  • How does the 'Switchback' (time-based) randomization design compare to your proposed geo-cluster design in terms of bias and variance?