Katkat
Back

Run EDA and train models while preventing overfitting

ImcData Scientist · Onsite
MediumData ScientistOnsiteMachine Learning

Last updated

This problem tests your end-to-end machine learning pipeline skills, focusing on tabular regression, feature engineering, validation strategies, and regularization techniques.

Scenario

You are tasked with building a regression model to predict housing prices in a new market using a dataset of 50,000 observations and 20 features (a mix of numerical and categorical). The goal is to minimize Mean Absolute Error (MAE) while ensuring the model generalizes well to unseen data.

Tasks

  1. Exploratory Data Analysis (EDA): Describe how you would identify data leakage and feature importance early in the process. Which visualization or statistical test would you use to detect multi-collinearity?
  2. Preprocessing: Propose a pipeline for handling missing values and high-cardinality categorical variables.
  3. Modeling: You are asked to compare a Lasso Regression (Linear) model with a shallow Multi-Layer Perceptron (MLP). Explain your feature scaling choices for each model.
  4. Overfitting Mitigation: If your MLP performs significantly better on the training set than the validation set, what specific architectural or training changes would you implement to reduce the generalization gap?

Requirements

  • Provide a clear validation strategy for this dataset (e.g., cross-validation vs. hold-out).
  • Explain how you would evaluate the final model's performance beyond just the MAE metric.

Example

Assume one feature is 'YearBuilt'. Input: Training set mean error = 5,000; Validation set mean error = 25,000. Output: Diagnosis of overfitting and a proposed fix, such as adding Dropout layers, implementing L2 regularization, or performing early stopping.

Hints

Hint 1

Consider the difference between global feature importance (like correlation matrices) and local importance methods like SHAP or permutation importance.

Hint 2

Think about why tree-based models might be preferred over neural networks for small-to-medium tabular datasets and how they handle non-linear relationships differently.

Hint 3

Discuss the 'Bias-Variance Trade-off' in the context of increasing MLP complexity or adding too many features.

Follow-up questions

  • How would your approach change if the dataset were imbalanced or had a significant number of outliers in the target variable?
  • If the model is deployed and performance degrades over time (data drift), how would you detect it and trigger a model update?
  • Explain how you would handle an interaction effect where the impact of 'SquareFootage' on price depends heavily on the 'Neighborhood' category.