Katkat
Back

Determine Sample Size for Promotion Campaign A/B Test

UberData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Determine Sample Size for Promotion Campaign A/B Test states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Scenario

Uber plans to launch a promotion campaign and wants to evaluate its effectiveness with an A/B experiment. Two concrete variants of the offer have come up in interviews:

  • A spend-threshold discount (e.g., 20% off when a user spends $40), and
  • A geo-targeted promo limited to a single market (e.g., a January-2024 campaign available only to San-Francisco users).

Treat the offer as a generic promotion; the reasoning below applies to either framing.

Question
  1. Business rationale. Why might the company want to launch this promotion campaign? (If geo-limited, why restrict it to a single market like San Francisco?)
  2. Success metrics. Which business metrics would you monitor to judge success? Distinguish primary, secondary/diagnostic, and guardrail metrics.
  3. Sample size. How would you determine the required sample size for the study? What inputs, assumptions, and factors influence it?
  4. Interpreting the MDE. Revenue is chosen as the primary metric and the product manager sets the minimum detectable effect (MDE) to 0.5%. How do you interpret this value and how does it affect the sample-size calculation?
  5. Non-significant result. Suppose the observed lift is +0.3% but the result is not statistically significant. How would you interpret this outcome and communicate it to the product manager?
Hints

Think about goal alignment, primary/secondary/guardrail KPIs, baseline mean and variance, power and Type-I/Type-II errors, effect size, significance level, variance reduction (e.g., CUPED), and how to explain a small, directional-but-non-significant effect to a stakeholder.