Katkat
Back

Interpret Model Metrics, Regularization, and Resampling Fundamentals

J.P. MorganData Scientist · HR Screen
MediumData ScientistHR ScreenStatistics & Math

Last updated

This problem tests your understanding of foundational machine learning metrics for imbalanced classification and the theoretical utility of regularization and resampling techniques.

Part 1: Metrics for Imbalanced Data

You are building a fraud detection model where the positive class (fraud) represents only 0.1% of the total dataset.

  1. Explain why Accuracy is an insufficient metric in this scenario.
  2. Define 'Lift' at a specific decile and explain how it differs from Precision.
  3. If Cohen’s Kappa is used to evaluate this model, explain what the 'expected agreement' term represents in the context of your imbalanced data.

Part 2: Regularization and Resampling

Consider a linear regression model with p features and n observations where p > n.

  1. Explain how L1 (Lasso) and L2 (Ridge) regularization differ in their effect on feature weights. Which is more likely to lead to a sparse model, and why?
  2. Compare Bagging (Bootstrap Aggregating) to standard model training. Specifically, explain how Bagging reduces variance while maintaining the bias of the base estimator.
  3. If you have a fair coin, what is the probability of getting exactly 7 heads in 10 flips? Provide the setup for the binomial calculation.

Hints

Hint 1

Consider the accuracy paradox: if a model predicts 'never fraud,' it achieves 99.9% accuracy but fails its primary objective.

Hint 2

Recall that L1 regularization introduces a Laplace prior on coefficients, forcing coefficients exactly to zero, while L2 introduces a Gaussian prior.

Hint 3

Bagging relies on the principle that the average of multiple independent, identically distributed (i.i.d.) variables reduces variance by a factor of 1/N, assuming the base models are decorrelated.

Follow-up questions

  • How would you adjust the classification threshold if the cost of a false negative is 10 times higher than the cost of a false positive?
  • If you observe that your model's training error is low but testing error is high, which form of regularization would you increase, and what happens to the bias-variance trade-off?
  • Explain the relationship between the Adjusted R-squared and the number of predictors in a linear model—why does it penalize the addition of useless features?