Determine A/B test sample size drivers
TubiData Scientist · Technical ScreenLast updated
This question evaluates statistical power and sample size determination for A/B testing, covering hypothesis testing for proportions, derivation using normal approximations, the impact of MDE, α and power choices, allocation ratios, cluster design (ICC and design effect), group‑sequential monitoring, and adjustment for overdispersed count outcomes.
For an A/B test with baseline conversion p0=8.0%, you want to detect a relative MDE of 8% (i.e., absolute +0.64 pp) at two‑sided alpha=0.05 and 80% power with 1:1 allocation. (a) Derive and compute the required per‑arm sample size using a normal approximation to the difference in proportions (show the z terms you use). (b) Now, holding everything else constant, state qualitatively and quantitatively how the required n changes when: MDE halves; alpha is tightened to 0.01; power increases to 90%; allocation is 75/25; variance is higher due to user‑level clustering with ICC=0.02 and average cluster size m=5 (compute the design effect); and you adopt group‑sequential monitoring with two equally spaced looks using O’Brien‑Fleming boundaries. (c) If the metric is overdispersed count data (NB2), outline how you would re‑estimate n.
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.