Katkat

Compute A/B sample size under clustering

UberData Scientist · Technical Screen

Last updated

This question evaluates competency in experimental design and statistical inference for A/B testing, specifically sample size calculation for two-sample proportions with unequal allocation, clustering effects (ICC and design effect), and adjustments for bot removal and data-quality attrition.

Compute the required sample size per variant for a two-arm signup A/B test with the following: baseline conversion p0 = 6%, target relative uplift = +5% (MDE = 0.05 × p0), two-sided α = 0.05, power = 0.80, traffic split 2:1 (control:treatment), 15% of sessions are bots removed post-hoc, sessions cluster by user (mean 1.4 sessions/user, ICC = 0.03), and 8% expected attrition from data quality filters. Provide formulas for proportions tests, apply the design effect for clustering, adjust for attrition and allocation ratio, and convert the result to test duration given 120,000 sessions/day. State any additional assumptions and how violations (variance mis-specification, sequential peeking) would change the plan.