Katkat

Differentiate and control Type I/II errors

OneMain FinancialData Scientist · Technical Screen

Last updated

This question evaluates understanding of hypothesis testing fundamentals such as Type I/Type II errors, statistical power and sample size calculation for two-proportion tests, family-wise error rate control (e.g., Bonferroni), and sequential analysis/alpha-spending schemes.

You are planning a two-arm A/B test on sign-up conversion. Baseline conversion p0 = 0.10. You want to detect an absolute uplift of +0.01 (to 0.11) with a two-sided α = 0.05 and power 1−β = 0.80 using a z-test for two proportions. 1) Derive the required sample size per arm and show your formula and numbers. 2) If you will also monitor 10 secondary metrics and must control the family-wise error rate with Bonferroni, what is the per-metric α and how does this change your sample size? 3) If you plan to peek daily over 14 days, describe a valid sequential scheme (e.g., O’Brien–Fleming or Pocock alpha-spending) and how it alters the stopping boundary. 4) Explain the practical cost of Type II error in this test and give one concrete way to reduce β without increasing α (quantify the trade-off).