Test two models' proportions for significance
MetaData Scientist · OnsiteLast updated
This question evaluates proficiency in statistical inference for proportions, covering hypothesis testing, confidence intervals, power and sample-size calculations, multiple-testing correction, and comparison of frequentist versus Bayesian approaches within the Statistics & Math domain for data scientist roles.
Two search models, A and B, were each used once by 100 distinct users (one query per user). Success is defined per query by your composite metric (success=1, failure=0). Model A had 90 successes, Model B had 85. Using a two-sided test at alpha=0.05: 1) State H0 and H1, choose the appropriate test (pooled two-proportion z-test), compute the test statistic and p-value, and conclude whether A outperforms B. 2) Compute a 95% confidence interval for pA−pB and interpret it for practical significance. 3) What per-arm sample size is needed to detect a +5 percentage-point uplift (baseline 85%) with 80% power at alpha=0.05? Show formulas/inputs. 4) If you simultaneously test these two models across 10 independent intents, apply a Bonferroni correction and say whether your conclusion changes. 5) Briefly explain when you would prefer an exact test or a Bayesian comparison and what you would report in each case.
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.