Compute sample size and test duration correctly
MetaData Scientist · Technical ScreenLast updated
This question evaluates competency in experiment design and statistical power calculations, including sample size estimation, variance misspecification effects, multiple-comparison corrections, sequential monitoring boundaries, and adjustments for covariates or clustering.
Compute sample size and duration for two experiments and defend your assumptions. Scenario A: Baseline friend-accept rate = 9.0%; target MDE = +0.5 percentage points (absolute); two-sided α=0.05, power=0.80, equal allocation; eligible traffic = 10M users/day. 1) Derive the per-arm sample size and calendar days required; 2) show how results change if the true SD is 20% higher than assumed. Scenario B (three arms, shared control): detect +0.3 pp for either treatment vs control with familywise α=0.05; choose and justify Holm vs Bonferroni vs Dunnett; recompute per-arm n and total duration with 20% traffic to control, 40% each to two treatments. For both scenarios: 3) discuss consequences of daily peeking with a naive p<0.05 rule; propose a sequential design (e.g., O’Brien–Fleming) and how it changes stopping boundaries; 4) explain when you’d switch to user-level CUPED or cluster-robust SEs (e.g., feed-level clustering) and how that affects n.
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.