Katkat

Determine A/B test sample size drivers

TubiData Scientist · Technical Screen

Last updated

This question evaluates statistical power and sample size determination for A/B testing, covering hypothesis testing for proportions, derivation using normal approximations, the impact of MDE, α and power choices, allocation ratios, cluster design (ICC and design effect), group‑sequential monitoring, and adjustment for overdispersed count outcomes.

For an A/B test with baseline conversion p0=8.0%, you want to detect a relative MDE of 8% (i.e., absolute +0.64 pp) at two‑sided alpha=0.05 and 80% power with 1:1 allocation. (a) Derive and compute the required per‑arm sample size using a normal approximation to the difference in proportions (show the z terms you use). (b) Now, holding everything else constant, state qualitatively and quantitatively how the required n changes when: MDE halves; alpha is tightened to 0.01; power increases to 90%; allocation is 75/25; variance is higher due to user‑level clustering with ICC=0.02 and average cluster size m=5 (compute the design effect); and you adopt group‑sequential monitoring with two equally spaced looks using O’Brien‑Fleming boundaries. (c) If the metric is overdispersed count data (NB2), outline how you would re‑estimate n.