Katkat
Back

Estimate Experiment Duration with Power Analysis

YelpData Scientist · Technical Screen
EasyData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests the ability to perform a power analysis to calculate sample size requirements for an A/B test and derive the experiment duration based on traffic constraints.

Scenario

At a local restaurant review platform, you are tasked with launching an experiment to test a new "Book Table" button color. You are tracking the Click-Through Rate (CTR) for this button.

Data & Assumptions

  • Current baseline CTR: 5.0%
  • Minimum Detectable Effect (MDE): A relative increase of 10% (i.e., you want to detect an increase from 5.0% to 5.5%).
  • Significance Level (alpha): 0.05
  • Statistical Power (1 - beta): 0.80
  • Daily traffic eligible for the experiment: 20,000 users per day.
  • The experiment will split traffic 50/50 between the control and variant groups.

Task

  1. Calculate the required sample size per group using the standard formula for a two-proportion z-test. You may approximate the standard deviation as sqrt(p * (1-p)), where p is the pooled probability.
  2. Given the daily traffic, estimate the total number of days required to reach the necessary sample size.
  3. Discuss what happens to your required sample size if you decide to decrease the MDE (make it harder to detect smaller changes).

Constraints

  • Assume a normal distribution approximation.
  • Round your final sample size up to the nearest whole user.

Hints

Hint 1

Recall the formula for sample size: `n = (Z_alpha/2 + Z_beta)^2 * (p1(1-p1) + p2(1-p2)) / (p1 - p2)^2`.

Hint 2

For alpha = 0.05 and power = 0.80, the sum of Z-scores (Z_alpha/2 + Z_beta) is approximately 2.8.

Hint 3

Calculate the sample size per group first, then divide by the number of daily users assigned to each group to get the duration.

Follow-up questions

  • How would your required sample size change if you increased the statistical power from 0.80 to 0.90?
  • If you discovered that 20% of your users were being incorrectly assigned to the variant group (a bias), how would that impact your experimental results?
  • Beyond sample size, what external factors (e.g., day-of-week effects, holidays) might force you to extend the experiment duration beyond your mathematical calculation?