Katkat
Back

Diagnose and fix flawed model fit

GoogleData Scientist · Technical Screen
HardData ScientistTechnical ScreenMachine Learning

Last updated

This question tests your ability to diagnose poor model performance in a binary classification task involving imbalanced data, high-cardinality categorical features, and potential bias.

Scenario

You are developing a binary classification model for a content moderation system designed to flag policy-violating images. You have a dataset of 1 million images with a target variable is_violation (1 for violation, 0 for safe). The violation rate is 0.5%.

You implemented a Logistic Regression model using raw pixel intensities as features and a subset of user-generated metadata (e.g., upload_location, device_id).

The Problem

Your initial results are concerning:

  1. The model reports 99.5% accuracy but catches zero violations.
  2. You notice that features like device_id have thousands of unique values, leading to extremely sparse input vectors.
  3. The business team mentions that the model seems to perform significantly worse on images uploaded from specific regions.

Your Task

Describe a comprehensive plan to diagnose and improve this model. Your response should address:

  1. How you would fix the zero-recall issue and choose a more appropriate evaluation metric.
  2. How you would handle the high-cardinality categorical variables like device_id.
  3. How you would investigate and mitigate the performance disparity across regions.
  4. How you would approach the scaling and transformation of features to ensure the Logistic Regression converges correctly.

Hints

Hint 1

Consider the difference between accuracy and metrics like precision-recall curves or F1-score when dealing with extreme class imbalance.

Hint 2

Think about dimensionality reduction techniques for categorical features, such as target encoding or embeddings, instead of one-hot encoding.

Hint 3

Analyze the loss function; would weighting the minority class differently or using a different probability threshold shift the decision boundary effectively?

Follow-up questions

  • How would you determine if the performance disparity by region is due to inherent data bias or a feature-level proxy variable?
  • If we move from Logistic Regression to a Gradient Boosted Decision Tree (e.g., LightGBM), how does that change your approach to feature preprocessing?
  • How would you calibrate the final predicted probabilities if the business needs to present a 'confidence score' to human moderators?