Design a leak-free time-split model
StripeData Scientist · Technical ScreenLast updated
This question evaluates a Data Scientist's competency in time-aware predictive modeling, including label leakage prevention for late-arriving events, temporal cross-validation, feature engineering, model selection (logistic regression vs gradient-boosted trees), calibration and thresholding for costed actions.
You have one week (recommended 4–6 hours of effort) to deliver slides plus runnable code that predicts each active user's probability of making a purchase in the next 30 days as of a snapshot time. Requirements: (1) Define snapshot_ts = 2025-09-01 00:00:00 UTC; predict whether an order occurs in [snapshot_ts, snapshot_ts + 30d). (2) Specify exactly how you will prevent label leakage, including late-arriving events (events whose arrival_time > event_time) and any features computed after snapshot_ts. (3) Propose a minimal but strong baseline and a main model, justify choosing between logistic regression with monotonic/regularized features versus gradient-boosted trees; list the top 5 engineered features you would start with and why. (4) Choose the primary optimization metric under class imbalance (e.g., PR-AUC vs ROC-AUC) and the business metric you will report on slides; explain trade-offs. (5) Describe a temporal cross-validation scheme (rolling-origin/blocked) that produces an honest estimate; include how you will tune hyperparameters quickly without overfitting given the time budget. (6) Explain how you will detect and mitigate data/label leakage, target leakage, and train-test contamination; include at least two concrete checks you would code. (7) Outline your calibration plan (e.g., Platt scaling vs isotonic), threshold selection for an email targeting use-case with a cost per send, and how you'll communicate calibration and expected impact in slides. (8) Include a minimal ablation plan you would run within the timebox (which features/modeling choices you would drop first if time runs short), and the exact contents of 5–7 slide headlines that tell a compelling story to a hiring manager.
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.