Katkat
Back

Design promo experiment and explain correlation

UberData Scientist · Technical Screen
HardData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This problem tests your ability to distinguish correlation from causation in observational data and to design a robust A/B test to measure the incremental impact of a new rider promotion.

Scenario

Uber is considering a '50% off your next ride' promotion to increase rider engagement. Data from a previous observational study shows that riders who received a similar promo code in the past had 15% higher spend over the following month compared to those who did not. However, the product team is skeptical that this uplift is purely causal.

Part 1: Causal Inference

  1. Identify two potential confounding variables that could explain the observed 15% correlation between promo usage and increased spend.
  2. Explain why a naive comparison of 'promo users' vs 'non-promo users' likely overestimates the true incremental impact (the Average Treatment Effect).

Part 2: Experiment Design

Design a randomized experiment to measure the causal impact of this promotion on Total Gross Bookings. Your response must include:

  • Unit of Randomization: Define the entity you will randomize and justify why.
  • Metric Definition: Define a primary metric and a secondary 'guardrail' metric to monitor for negative side effects.
  • Interference: In a two-sided marketplace, a promotion might cause 'cannibalization' or spillover effects. Explain what network interference is in this context and how you would mitigate it (e.g., cluster randomization or switchback design).
  • Sample Size: Briefly describe the factors that determine the required sample size for this test.

Hints

Hint 1

Consider that promo codes are often sent to the most active users who would have booked rides regardless of the discount.

Hint 2

Think about the impact of the promotion on driver availability and surge pricing, which affects users who did not receive the promo.

Hint 3

Consider using geographic regions or distinct time buckets (switchback) to isolate the treatment effect from marketplace-wide dynamics.

Follow-up questions

  • How would you measure the 'cannibalization' effect, where the promo increases short-term volume but decreases long-term margins?
  • If you observe that riders in the treatment group are booking shorter, less profitable rides, how would you adjust your primary metric?
  • What statistical methods would you use to account for the fact that some users in the control group might see ads or hear about the promo from friends (SUTVA violation)?