Katkat
Back

Evaluate marketplace interventions

UberData Scientist · Onsite
MediumData ScientistOnsiteAnalytics & Experimentation

Last updated

This question tests the ability to design an experimentation framework for a two-sided marketplace, specifically addressing interference between supply and demand and selecting appropriate success metrics.

Scenario

Uber is considering introducing a new feature called "Priority Matching," which allows riders to pay an extra fee to be prioritized in the dispatch queue. You are tasked with designing an experiment to measure the impact of this feature on overall marketplace health.

The Challenge

  1. Metric Selection: Define one primary metric for the driver side and one primary metric for the rider side to measure the impact of this feature. Include one guardrail metric to ensure the feature does not degrade the marketplace.
  2. Experiment Design: The team suggests a simple A/B test by randomizing users (riders) into treatment and control groups. Explain why this approach might lead to biased results in a marketplace setting. What is the phenomenon called, and how does it affect your estimates?
  3. Methodology: Propose a better randomization strategy to minimize the bias identified in step 2. If you cannot use a perfect randomization strategy, what statistical techniques or observational methods would you employ to estimate the causal impact?

Example Context

  • Marketplace: Two-sided (Riders/Drivers).
  • Constraints: High churn risk for drivers if wait times increase; high sensitivity of riders to surge pricing.

Hints

Hint 1

Consider that the treatment of one user group (e.g., riders getting priority) directly impacts the availability and wait times of the other group (e.g., drivers or non-priority riders).

Hint 2

Think about 'network effects' or 'interference' where the Stable Unit Treatment Value Assumption (SUTVA) is violated.

Hint 3

Explore options like cluster-based randomization (e.g., by city or time-slice) or synthetic control methods to mitigate interference.

Follow-up questions

  • How would you account for the 'cannibalization' effect where priority riders simply shift demand from non-priority sessions?
  • If the experiment shows an increase in revenue but a decrease in long-term rider retention, how would you weight these metrics to make a launch decision?
  • What steps would you take to ensure that driver earnings are not negatively impacted by the dispatch logic changes?