Katkat
Back

Resolve a Mismatch Between Regression and Classification Metrics

Capital OneData Scientist · Online Assessment
MediumData ScientistOnline AssessmentMachine Learning

Last updated

Tests your ability to diagnose modeling and metric mismatches in credit risk and fraud scenarios, design proper two-stage architectures, and select evaluation metrics aligned with business decisions.

Scenario

A junior modeler on your risk modeling team built an end-to-end pipeline to predict credit card charge-off losses. The target variable is loss_amount, representing the net dollar amount charged off within 12 months after account opening.

Historically:

  • 97.5% of accounts incur loss_amount = $0.00 (good standing or repaid).
  • 2.5% of accounts default, with loss_amount exhibiting a heavy right skew ranging from $150 to $35,000 (mean: $3,200, median: $1,800).

The previous modeler trained a single LightGBM regressor minimizing Root Mean Squared Error (RMSE) directly on loss_amount. However, to evaluate model quality, they thresholded predictions at loss_amount > $500 and reported standard binary metrics: Precision = 0.22, Recall = 0.81, and ROC-AUC = 0.89.

Leadership is confused by the conflicting signals: RMSE looks low due to the dominance of zero-loss accounts, yet the classification precision at the chosen threshold is poor, and capital reserves require dollar-based accuracy rather than binary labels.

Your Task

  1. Diagnose the Mismatch:

    • Explain why optimizing RMSE on zero-inflated continuous targets produces suboptimal results for both decision-making and downstream binary ranking.
    • Critique the validity and risks of using classification metrics (Precision, Recall, ROC-AUC) on thresholded continuous regression outputs without re-calibrating probabilities.

  2. Formulate a Two-Stage Modeling Architecture:

    • Propose an alternative modeling framework (e.g., Hurdle or Two-Stage Model). Detail Stage 1 (classification) and Stage 2 (conditional regression), including their respective loss functions, training sets, and output representations.
    • Show how the final expected loss E[loss | X] is derived from the components.

  3. Define Decision-Aligned Metrics:

    • Recommend specific evaluation metrics for Stage 1, Stage 2, and the combined expected loss.
    • Define a business utility metric (e.g., Expected Value or Profit Curve) that accounts for asymmetric costs: the cost of a false positive (declining a profitable customer) versus a false negative (accepting an account that charges off).

Hints

Hint 1

Consider how quadratic loss (RMSE) behaves when 97.5% of values are zero while a few outliers are in the tens of thousands of dollars.

Hint 2

A two-stage approach typically decouples P(Default = 1 | X) from E[Loss | Default = 1, X], avoiding the zero-inflation distortion in the regression stage.

Hint 3

Tie binary decision thresholds to the cost ratio: a dollar saved on fraud/loss prevention versus lost revenue from rejected creditworthy customers.

Follow-up questions

  • How would you handle sample selection bias in Stage 2 since the conditional loss is only observed on defaulted accounts?
  • If business constraints demand a single end-to-end model instead of a two-stage pipeline, what loss function (e.g., Tweedie deviance) would you use and why?
  • How does threshold tuning on predicted probabilities change if interest rates, interchange fees, and credit limits vary dynamically per applicant?