Katkat
Back

Test classifier difference with McNemar's test

MicrosoftData Scientist · Onsite
MediumData ScientistOnsiteStatistics & Math

Last updated

This question tests a data scientist's ability to apply McNemar's test for comparing paired classifiers, a core competency in statistical hypothesis testing and model evaluation. It assesses practical knowledge of contingency table analysis, exact vs. asymptotic tests, confidence interval construction, and multiple testing correction in machine learning contexts.

You evaluated two classifiers A and B on the SAME 10,000 labeled examples. The paired outcomes are:

  • Both correct n11 = 8,740; both wrong n00 = 740; A correct/B wrong n10 = 300; A wrong/B correct n01 = 220. Answer:
  1. Using McNemar's test with continuity correction, compute the test statistic and p-value for H0: error rates are equal. Show intermediate numbers (b, c, |b−c|, b+c).
  2. Compute the exact binomial p-value for the same H0 using b+c trials. Explain when you prefer the exact test.
  3. Give a 95% confidence interval for the accuracy difference (A−B) on paired data; state which method you use and why.
  4. Discuss assumptions, when McNemar's test is inappropriate, and how you'd adjust if you compare A against 10 models (multiple testing control).