Katkat
Back

Design and Interpret an A/B Test

InstacartData Scientist · Onsite
HardData ScientistOnsiteAnalytics & Experimentation

Last updated

This question tests the ability to design a statistically rigorous A/B test, calculate sample sizes based on effect thresholds, and account for real-world constraints like seasonality and multiple comparisons.

Scenario

Instacart is considering a change to the "Recommended for You" algorithm. The product team wants to test if a new ranking model increases the Average Order Value (AOV).

  • Baseline AOV: $100
  • Minimum Detectable Effect (MDE): 2% ($2 increase)
  • Standard Deviation of AOV: $40
  • Daily Active Users (DAU): 500,000
  • Expected exposure rate: 20% of users will be eligible for the experiment.
  • Significance level (alpha): 0.05
  • Power (1-beta): 0.80

Task

  1. Calculate the required sample size per variant for this experiment. (You may assume a standard normal distribution approach where Z_alpha/2 = 1.96 and Z_beta = 0.84).
  2. Given the traffic constraints, how long should the experiment run to reach significance?
  3. The team is worried about "novelty effects." How would you distinguish between a temporary novelty effect and a permanent shift in user behavior?
  4. Suppose the team decides to check the p-value every day and stop as soon as it drops below 0.05. Explain the statistical risks associated with this strategy and how to mitigate them.

Hints

Hint 1

Recall the formula for sample size per group: n = 2 * sigma^2 * (Z_alpha/2 + Z_beta)^2 / (delta)^2.

Hint 2

Calculate the number of users entering the experiment daily and divide your total required sample size by this daily inflow.

Hint 3

Consider the impact of 'peeking' at data on the False Positive rate and how Sequential Analysis or fixed-horizon testing resolves this.

Follow-up questions

  • How would you handle a situation where the new algorithm only affects a specific subset of users, like those who order more than 5 times per month?
  • If the AOV increases but the total number of orders decreases, how would you frame the trade-off to stakeholders?
  • How does the choice of a one-tailed vs. two-tailed test affect your sample size calculation and the interpretability of your results?