Katkat
Back

Use confusion matrix to choose model metric

MicrosoftData Scientist · Technical Screen
EasyData ScientistTechnical ScreenStatistics & Math

Last updated

This question tests the ability to select appropriate classification metrics based on specific business objectives, asymmetric error costs, and dataset characteristics like class imbalance.

Scenario

You are a Data Scientist working on a fraud detection system for a high-frequency payment processor. The system predicts whether a transaction is 'Fraudulent' or 'Legitimate'.

The Data

  • Total transactions per day: 1,000,000
  • Fraudulent transactions (Positive class): 100 (0.01%)
  • Legitimate transactions (Negative class): 999,900 (99.99%)

The Business Constraints

  • If the model flags a 'Legitimate' transaction as 'Fraudulent' (False Positive), the customer is blocked, which leads to a direct revenue loss and high customer churn.
  • If the model misses a 'Fraudulent' transaction (False Negative), the company loses the full value of the transaction plus processing fees.
  • The cost of a False Negative is estimated to be 10x higher than the cost of a False Positive.

Task

  1. Explain why 'Accuracy' is a misleading metric for this model and recommend a more suitable metric or set of metrics.
  2. Given the asymmetric cost structure, how would you adjust the decision threshold (typically 0.5) to optimize for business value? How would you validate this change?
  3. Would you prefer an ROC-AUC or a Precision-Recall (PR) AUC curve to evaluate this model? Justify your choice based on the class distribution.

Hints

Hint 1

Consider how a model that predicts 'Legitimate' for every transaction would perform in terms of accuracy given the class imbalance.

Hint 2

Think about the trade-off between Precision and Recall and how shifting the classification threshold moves the model along the PR curve.

Hint 3

Remember that ROC-AUC is sensitive to the number of True Negatives, which are extremely abundant in this dataset.

Follow-up questions

  • How would you address the class imbalance during the training phase before even considering evaluation metrics?
  • If the company suddenly changes its business model such that False Positives become more expensive than False Negatives, how should the model deployment strategy evolve?
  • What is the role of probability calibration in this scenario, and how would you assess if your model's predicted probabilities are reliable?