Katkat
Back

Train and improve a scikit-learn binary classifier

PerplexityMachine Learning Engineer · Technical Screen
MediumMachine Learning EngineerTechnical ScreenMachine Learning

Last updated

This problem tests your proficiency in the scikit-learn workflow, specifically focusing on data preprocessing, addressing class imbalance in imbalanced datasets, and selecting appropriate metrics beyond accuracy.

Problem Statement

You are tasked with building a binary classifier for an internal churn prediction system. The dataset consists of 100,000 user activity logs. The target variable is_churned is highly imbalanced, with only 5% of users marked as positive.

Your task is to write a Python script using scikit-learn that performs the following:

  1. Preprocessing: Apply StandardScaler to numerical features and OneHotEncoder to categorical features.
  2. Pipeline Construction: Create a pipeline that handles preprocessing and the classifier (start with a RandomForestClassifier).
  3. Evaluation: Implement a cross-validation strategy that respects the class imbalance (e.g., using StratifiedKFold).
  4. Optimization: Since accuracy is misleading here, configure the model to optimize for the Area Under the Precision-Recall Curve (AUPRC).

Requirements

  • Provide the code snippet for the pipeline.
  • Explain how you would handle the imbalance using either class_weight parameters or resampling techniques (e.g., SMOTE) within the pipeline.
  • Define the evaluation metric and justify why it is superior to Accuracy in this specific case.

Example Input/Output

Assume X is a DataFrame with columns ['age', 'tenure', 'region', 'monthly_spend'] and y is the binary target.

# Example Interface
from sklearn.pipeline import Pipeline

def build_churn_model(X, y):
    # Return a trained sklearn pipeline
    pass

Constraints

  • Use sklearn.compose.ColumnTransformer to handle different feature types.
  • Ensure the solution avoids data leakage during preprocessing.

Hints

Hint 1

Consider using `make_column_transformer` to easily map specific preprocessing steps to specific column types.

Hint 2

Think about whether oversampling (like SMOTE) should be applied to the entire dataset or only within the cross-validation folds.

Hint 3

When optimizing for AUPRC, remember that `RandomForestClassifier` allows you to pass `class_weight='balanced_subsample'` to handle minority class importance.

Follow-up questions

  • How would you interpret the feature importances if your model uses a pipeline with one-hot encoding?
  • If the business requires a 90% recall rate, how would you adjust the model's decision threshold after training?
  • How does the choice of AUPRC differ from using the ROC-AUC score in the context of highly imbalanced datasets?