Katkat
Back

Calculate A/B sample size, CI, decision rules

AmazonData Scientist · Onsite
MediumData ScientistOnsiteStatistics & Math

Last updated

This question evaluates skills in experimental design and statistical inference, including sample size calculation, hypothesis testing, confidence interval interpretation, multiple comparison control, variance reduction techniques, and operational considerations for A/B testing.

Design and analyze an A/B test for a signup funnel. Setup: Baseline conversion p0 = 0.040. Minimum detectable effect (absolute) = 0.004 (i.e., target p1 = 0.044). Two-sided α = 0.05, power = 0.80.

  1. Sample size: Compute required sample size per arm for a two-proportion z-test. Show the formula you use and the numeric result.
  2. CI and p-value: Suppose after launch you observe Control nC=50,000, xC=2,150 (4.30%) and Treatment nT=49,500, xT=2,090 (4.22%). Compute the p-value for the difference in proportions (both pooled and unpooled variants) and a 95% CI for (pT - pC). Interpret practically.
  3. Design choices: Specify primary vs. guardrail metrics and how you would correct for multiple comparisons if you track cancellation rate and latency as guardrails.
  4. Power/variance reduction: Explain how you’d use CUPED or stratification to reduce variance and how that changes required sample size.
  5. Operational risks: Discuss sequential peeking and how to control type-I error (e.g., alpha-spending). What do you do if traffic is heterogeneous and the global average obscures segment-level reversals?