How would you build and evaluate a classifier?
MicrosoftData Scientist · Technical ScreenLast updated
This problem tests your ability to design a binary classification pipeline from metric selection to production monitoring, with a focus on aligning technical decisions with business objectives.
Scenario: Fraud Detection System
You are tasked with building a binary classifier to detect fraudulent credit card transactions for a global banking platform.
Requirements
- Data Context: The dataset contains 10 million transactions per day, with a fraud rate of approximately 0.05%. The data is heavily imbalanced.
- Business Goal: You must minimize financial loss due to fraud while keeping the 'False Positive' rate low to avoid annoying legitimate customers.
Task
Design the end-to-end approach to solving this problem by addressing the following:
- Evaluation Metrics: Given the class imbalance, explain why Accuracy is an inappropriate metric and justify the selection of your primary optimization metric. How would you choose the classification threshold?
- Model Development: Describe your strategy for handling class imbalance (e.g., resampling, cost-sensitive learning) and feature engineering steps for transaction data.
- Deployment & Monitoring: How would you monitor the model's performance in production? Specifically, how do you handle 'model drift' when transaction patterns change?
- Interpretability: If a customer disputes a transaction marked as fraud, how would you use methods like SHAP to explain the model's decision, and what are the limitations of these explanations in a high-stakes financial context?
Expected Output
Provide a structured explanation covering your methodology for the points above, ensuring you highlight the trade-offs between precision and recall relative to the business goal.
Hints
Hint 1
Consider the cost asymmetry between missing a fraudulent transaction (lost funds) and flagging a legitimate one (customer churn).
Hint 2
Think about how the distribution of features might shift over time and what specific statistical tests or performance dashboards could capture this.
Hint 3
Reflect on how SHAP values explain individual predictions versus how they capture global model behavior.
Follow-up questions
- How would you adjust your model if the cost of a False Negative (fraudulent transaction) increases tenfold?
- If your model performs well on the test set but fails in production, what are the most likely reasons for this performance gap?
- How would you communicate the model's performance to non-technical stakeholders who are concerned about high false rejection rates?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.