Katkat

How would you build and evaluate a classifier?

MicrosoftData Scientist · Technical Screen

Last updated

This problem tests your ability to design a binary classification pipeline from metric selection to production monitoring, with a focus on aligning technical decisions with business objectives.

Scenario: Fraud Detection System

You are tasked with building a binary classifier to detect fraudulent credit card transactions for a global banking platform.

Requirements

  1. Data Context: The dataset contains 10 million transactions per day, with a fraud rate of approximately 0.05%. The data is heavily imbalanced.
  2. Business Goal: You must minimize financial loss due to fraud while keeping the 'False Positive' rate low to avoid annoying legitimate customers.

Task

Design the end-to-end approach to solving this problem by addressing the following:

  • Evaluation Metrics: Given the class imbalance, explain why Accuracy is an inappropriate metric and justify the selection of your primary optimization metric. How would you choose the classification threshold?
  • Model Development: Describe your strategy for handling class imbalance (e.g., resampling, cost-sensitive learning) and feature engineering steps for transaction data.
  • Deployment & Monitoring: How would you monitor the model's performance in production? Specifically, how do you handle 'model drift' when transaction patterns change?
  • Interpretability: If a customer disputes a transaction marked as fraud, how would you use methods like SHAP to explain the model's decision, and what are the limitations of these explanations in a high-stakes financial context?

Expected Output

Provide a structured explanation covering your methodology for the points above, ensuring you highlight the trade-offs between precision and recall relative to the business goal.

Hints

Hint 1

Consider the cost asymmetry between missing a fraudulent transaction (lost funds) and flagging a legitimate one (customer churn).

Hint 2

Think about how the distribution of features might shift over time and what specific statistical tests or performance dashboards could capture this.

Hint 3

Reflect on how SHAP values explain individual predictions versus how they capture global model behavior.

Follow-up questions

  • How would you adjust your model if the cost of a False Negative (fraudulent transaction) increases tenfold?
  • If your model performs well on the test set but fails in production, what are the most likely reasons for this performance gap?
  • How would you communicate the model's performance to non-technical stakeholders who are concerned about high false rejection rates?