Validate Features That Predict User Retention
MixpanelData Scientist · Technical ScreenLast updated
Learn to evaluate retention features with leakage-safe time windows, cohort validation, baselines, ablations, and calibrated predictions.
Validate Features That Predict User Retention
You have user records with signup time and plan type, plus timestamped user events. You are asked which features might predict retention and how to establish confidence in their predictive value. Candidate ideas include event count, active days, plan type, and signup weekday.
Describe an exploratory and modeling approach. Define the prediction time and retention window explicitly, explain how features are constructed without leakage, and show how you would test whether the features add useful signal. The retention definition is not fixed: make your chosen definition and assumptions explicit.
Hints
Hint 1
Draw the timeline: Ask whether every event used by a feature would already exist at the moment the prediction is made.
Follow-up questions
- What goes wrong if first-week event counts predict whether any event occurred in that same week?
- How would you interpret a feature that has high random-forest importance but adds little held-out performance?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.