Katkat
Back

How to experiment on ETA reduction

UberData Scientist · Technical Screen
EasyData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This problem tests the ability to design an A/B test for product improvements and address potential confounding factors like selection bias and spillover effects.

Scenario

Our product team at Uber is testing a new routing algorithm aimed at reducing the Estimated Time of Arrival (ETA) for riders. You are tasked with designing an experiment to determine if this new algorithm causes a meaningful decrease in ride requests.

The Problem

  1. Propose an experimental design to test the new routing algorithm. What is your primary metric, and how would you randomize users to ensure a valid causal inference?
  2. Suppose you observe that riders in the treatment group have lower conversion rates than the control group, despite the ETAs being lower. Brainstorm potential confounding factors or 'hidden' effects that could explain this observation.
  3. How would you handle potential 'spillover' or network interference effects, where the treatment given to one rider affects the supply of drivers available for others in the control group?

Constraints

  • Assume you have access to granular data on individual ride requests, driver locations, and historical conversion rates.
  • The system has two-sided market dynamics (riders and drivers).

Hints

Hint 1

Consider whether individual-level randomization is appropriate given that drivers serve multiple riders in a specific geographic area.

Hint 2

Think about the difference between 'intent to treat' and the actual realization of an ETA reduction, and how price changes or surge pricing might react to the new algorithm.

Hint 3

Evaluate whether a cluster-based randomization (e.g., by city or region) is safer than individual-level randomization to avoid network interference.

Follow-up questions

  • How would you determine the minimum detectable effect (MDE) and the required sample size for this experiment?
  • If the new algorithm improves ETAs but increases the number of driver cancellations, how would you balance these metrics in your final recommendation?
  • What statistical methods would you use to account for pre-experiment differences between treatment and control groups if randomization is imperfect?