Estimate CTR lift with binomial tests and errors
MetaData Scientist · OnsiteLast updated
This question evaluates statistical inference for proportions, sequential monitoring, and multiple-comparison error control—assessing competence with binomial-based CTR comparison, pooled standard errors, z-tests and confidence intervals, Type I error inflation from peeking, and FWER/FDR considerations.
In an A B test, control shows 10,000,000 impressions with 1.20% CTR; treatment shows 10,000,000 impressions with 1.23% CTR. (a) Compute the absolute and relative lift, the pooled standard error, z statistic, and two-sided p value. Provide a 95% confidence interval for the relative lift. (b) If you peeked daily over 14 days without alpha spending, quantify approximate Type I error inflation and propose a corrected sequential plan using alpha spending or a Bayesian alternative with a prior on lift. (c) If you also track 8 guardrail metrics, explain how you would control familywise error or false discovery rate.
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.