Katkat
Back

Choose Classification Metrics Under Asymmetric Costs

MicrosoftData Scientist · Technical Screen
MediumData ScientistTechnical ScreenStatistics & Math

Last updated

This question evaluates understanding of binary classification evaluation metrics and model-operating decisions, covering confusion-matrix-derived measures (precision, recall/sensitivity, specificity, false positive/negative rates), summary metrics (ROC-AUC, PR-AUC), Type I/II errors, threshold selection, prevalence drift, probability calibration, and cost-sensitive trade-offs. It is commonly asked to assess a data scientist's ability to reason about model performance under class imbalance and asymmetric business costs, and it belongs to the Statistics & Math / Machine Learning model-evaluation domain requiring both conceptual understanding and practical application.

You are evaluating a binary classification model for a business problem.

Explain how to use a confusion matrix to compute and interpret:

  • precision,
  • recall (sensitivity),
  • specificity,
  • false positive rate,
  • false negative rate,
  • ROC-AUC,
  • PR-AUC.

Also answer the following:

  1. What is the difference between Type I and Type II error?
  2. How does your metric choice change across scenarios such as spam filtering, medical screening, fraud detection, and ad click prediction?
  3. Why can accuracy be misleading under class imbalance?
  4. How do decision thresholds, prevalence drift, and probability calibration affect model evaluation?
  5. If false positives and false negatives have different business costs, how would you choose an operating threshold?