Detect leakage and evaluate a prediction model
MetaData Scientist · Onsite
Last updated
This question evaluates competence in production-ready supervised machine learning—detecting and repairing data leakage in logs-based features, designing time-aware cross-validation, defining KPI-driven thresholding and calibration, interpreting model coefficients, and establishing monitoring and retraining policies for a churn prediction model in the Machine Learning domain. It is commonly asked because interviewers need to assess both conceptual understanding of leakage, validation, and calibration principles and practical application skills for deploying and maintaining models with cost-sensitive objectives and monitoring in production, so the level of abstraction spans conceptual reasoning and hands-on practical application.
You inherit a churn prediction model trained to predict whether a user will place an order in the next 28 days. The training data include features computed weekly, some of which leak post-label information (e.g., features that include activity within the prediction window). (a) Identify at least five concrete leakage sources in a logs-based feature set (examples: features derived from orders within the label window, coupon_applied in the next 28 days, post-treatment delivery ETA, label-influenced support contacts). For each, rewrite the feature so it is computable at prediction time with a strict event-time cutoff. (b) Propose a time-based cross-validation scheme (rolling origin) and define train/validation/test splits ensuring no future leakage. Specify how you would handle users entering/leaving the cohort and cold-start users. (c) Offline metrics show AUC=0.79, PR-AUC=0.23. Online, targeting the top decile increases conversion by 1.5% but raises cancellations by 0.3pp. Define business KPIs and a cost-sensitive objective to tune the decision threshold; include calibration (Platt/Isotonic) and how you would check calibration drift in production. (d) Interpretation: a standardized feature past_7d_orders has logistic regression coefficient 0.40 and baseline log-odds of churn −1.50. Compute the odds ratio for a +1 SD increase and the resulting churn probability change from the baseline. Explain limitations of such ceteris paribus interpretations in correlated settings. (e) Outline a monitoring plan (data quality, feature distributions, PSI, label delay, prediction drift) and a retraining policy tied to performance and covariate shift triggers.
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.