Katkat

Estimate CTR lift with binomial tests and errors

MetaData Scientist · Onsite

Last updated

This question evaluates statistical inference for proportions, sequential monitoring, and multiple-comparison error control—assessing competence with binomial-based CTR comparison, pooled standard errors, z-tests and confidence intervals, Type I error inflation from peeking, and FWER/FDR considerations.

In an A B test, control shows 10,000,000 impressions with 1.20% CTR; treatment shows 10,000,000 impressions with 1.23% CTR. (a) Compute the absolute and relative lift, the pooled standard error, z statistic, and two-sided p value. Provide a 95% confidence interval for the relative lift. (b) If you peeked daily over 14 days without alpha spending, quantify approximate Type I error inflation and propose a corrected sequential plan using alpha spending or a Bayesian alternative with a prior on lift. (c) If you also track 8 guardrail metrics, explain how you would control familywise error or false discovery rate.