Katkat
Back

Build a leak-free sklearn churn pipeline

CVS HealthData Scientist · Online Assessment
MediumData ScientistOnline AssessmentMachine Learning

Last updated

This question evaluates practical competencies in building a reproducible scikit-learn churn prediction pipeline—covering temporal splitting to avoid leakage, preprocessing, calibration, hyperparameter tuning, evaluation with ROC AUC/PR AUC and F1-based thresholds, and permutation feature importance—and is in the Machine Learning domain.

Write Python (sklearn) code to build a reproducible classification pipeline for predicting user subscription in the next 30 days. Dataset columns: user_id (str), event_date (date), country (categorical), device_type (categorical), sessions_last_7d (int), purchases_last_30d (int), avg_session_secs (float), days_since_signup (int), is_subscribed (0/1 target). Requirements: a) Split the data temporally: use rows with event_date ≤ 2025-08-25 for training and 2025-08-26..2025-09-01 for validation ("today" is 2025-09-01). b) Use ColumnTransformer with: numeric pipeline = SimpleImputer(strategy='median') + StandardScaler(); categorical pipeline = SimpleImputer(strategy='most_frequent') + OneHotEncoder(handle_unknown='ignore'). c) Use LogisticRegression with class_weight='balanced' and max_iter tuned; perform a small hyperparameter search over C on a StratifiedKFold CV within the training set only. d) Evaluate ROC AUC and PR AUC on the validation window; also select a decision threshold that maximizes F1 on validation and report precision/recall/F1 at that threshold. e) Calibrate predicted probabilities using CalibratedClassifierCV on the training set (CV=3) without leaking validation. f) Demonstrate how you would compute permutation feature importance on the validation set and list the top 5 features by importance. g) Briefly explain one potential target leakage risk in this schema and how your pipeline avoids it.