Katkat
Back

Evaluate Marketplace Changes

UberData Scientist · Onsite
MediumData ScientistOnsiteAnalytics & Experimentation

Last updated

This problem tests your ability to design an A/B test for a two-sided marketplace, focusing on network effects, metric selection, and identifying potential interference between treatment and control groups.

Marketplace Experiment Design

Uber is planning to introduce 'Dynamic Surge Pricing' in a specific city to balance supply (drivers) and demand (riders) during peak hours. You are tasked with designing an experiment to measure the impact of this feature on the marketplace.

The Challenge

  1. Metric Definition: Define a primary metric for the experiment. Given the marketplace nature, explain how you would select guardrail metrics to monitor for unintended negative consequences.
  2. Randomization Strategy: You propose a randomized experiment. Would you randomize at the user level, the driver level, or the city/geographic level? Justify your choice, specifically addressing potential network effects or cross-group interference.
  3. Causal Inference: Suppose you choose to run this test as a geo-experiment (randomizing by city). How would you account for the fact that some cities are more 'business-heavy' while others are 'tourist-heavy' when interpreting the treatment effect?

Requirements

  • Provide a brief rationale for your chosen unit of randomization.
  • List at least three specific guardrail metrics and explain why each is necessary for a two-sided marketplace.
  • Discuss the trade-offs between a user-level randomization versus a cluster/geographic randomization in this specific context.

Hints

Hint 1

Consider whether an action taken by a driver in the treatment group affects the availability or pricing for a rider in the control group.

Hint 2

Think about 'interference'—if prices change for one user, how does that impact the probability of a driver being available for someone else?

Hint 3

For the geo-experiment, consider using a synthetic control method or a pre-experiment matching approach to ensure comparability across heterogeneous cities.

Follow-up questions

  • How would you handle the 'SUTVA' (Stable Unit Treatment Value Assumption) violation if drivers frequently switch between geographic zones during their shifts?
  • If the experiment results show a significant increase in rider completion rates but a decrease in driver hourly earnings, how would you decide whether to roll out the feature?
  • What statistical methods would you use to account for the temporal autocorrelation (e.g., peak hour spikes) inherent in ride-hailing data?