Katkat
Back

Build a leak-free sklearn pipeline

Boston Consulting GroupData Scientist · Online Assessment
MediumData ScientistOnline AssessmentMachine Learning

Last updated

This problem tests the ability to build a robust scikit-learn pipeline that prevents data leakage during preprocessing and hyperparameter tuning while handling class imbalance.

The Task

You are tasked with building a machine learning pipeline to predict customer churn based on a dataset containing numerical features (e.g., account balance) and categorical features (e.g., region). The target variable is imbalanced (only 5% churn).

Construct a scikit-learn Pipeline that performs the following steps in order:

  1. Preprocessing: Apply StandardScaler to numerical columns and OneHotEncoder to categorical columns using ColumnTransformer.
  2. Resampling: Use SMOTE (from imblearn) to handle class imbalance, ensuring it only triggers during training.
  3. Modeling: Use a RandomForestClassifier.

Finally, wrap this pipeline in a GridSearchCV object to tune max_depth for the forest and evaluate performance using StratifiedKFold (5 splits) with f1_macro as the scoring metric.

Requirements

  • Ensure the implementation is immune to data leakage (the scaler and resampler must not be fit on validation folds).
  • Use imblearn.pipeline.Pipeline instead of sklearn.pipeline.Pipeline to correctly handle resampling steps.

Example Workflow

Input: A pandas DataFrame df with X features and y target. Output: A trained GridSearchCV object.

from imblearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
# ... imports

def build_pipeline(num_cols, cat_cols):
    # Define your transformers, pipeline, and grid search here
    pass

Edge Cases to Consider

  • How do you handle unknown categories in the validation set during OneHotEncoder transformation?
  • Why is it critical to use imblearn's Pipeline class instead of the standard sklearn version when using SMOTE?

Hints

Hint 1

Standard scikit-learn pipelines do not allow for steps that change the number of samples, so use the pipeline module from the imbalanced-learn library.

Hint 2

Data leakage is avoided when you fit your transformers and resamplers strictly within the cross-validation loop, which `GridSearchCV` does automatically when given a `Pipeline` object.

Hint 3

Ensure `handle_unknown='ignore'` is set in your `OneHotEncoder` to prevent the pipeline from crashing on categories not seen during training.

Follow-up questions

  • How would you modify this pipeline if you needed to perform feature selection (e.g., SelectKBest) after the imputation/scaling steps?
  • If the model is deployed in a real-time production environment, how does the `Pipeline` object simplify the inference process compared to manual preprocessing?
  • Why is `f1_macro` or `balanced_accuracy` preferred over standard `accuracy` when dealing with the imbalanced dataset described?