Katkat
Back

Build Churn Prediction and Survival Models

GustoData Scientist · Technical Screen
MediumData ScientistTechnical ScreenMachine Learning

Last updated

This problem tests the ability to design an end-to-end churn prediction pipeline, specifically focusing on the distinction between binary classification and survival analysis in the context of customer retention.

Scenario

You are a Data Scientist at a subscription-based SaaS company. The product team wants to identify customers at risk of churning to launch targeted retention campaigns. Your dataset contains historical customer information, including: monthly_billing_amount, support_ticket_volume, days_since_last_login, subscription_start_date, and churn_status (binary).

Task

  1. Problem Formulation: Compare and contrast using a binary classifier (e.g., Random Forest or Logistic Regression) versus a survival model (e.g., Cox Proportional Hazards or Kaplan-Meier) for this task. What are the pros and cons regarding data handling and business utility?
  2. Feature Engineering & Leakage: Identify potential sources of data leakage when building your features. Specifically, how do you handle customers who have not churned yet (censored data) in your training set?
  3. Evaluation: If you choose a binary classifier, what metrics would you prioritize given the class imbalance (typically, only 2-5% of users churn per month)? How does this differ from evaluating a survival model like a Cox model (e.g., C-index)?
  4. Operationalization: How would you convert the model's output (probability vs. risk score) into a daily action list for the customer success team?

Constraints

  • Assume the data is longitudinal.
  • Consider the 'cold start' problem for new users versus 'tenured' users.
  • Discuss the trade-off between model interpretability (coefficients) and predictive performance.

Hints

Hint 1

Consider that binary classification ignores the temporal aspect of when a customer is likely to churn, whereas survival analysis models the probability of churn occurring over time.

Hint 2

Data leakage often occurs when including features that are calculated using information that wouldn't be available at the time of prediction, such as 'average support tickets per lifetime'.

Hint 3

For class imbalance, precision-recall curves or F1-scores are generally more informative than accuracy; for survival models, the C-index represents the model's ability to correctly order pairs of survival times.

Follow-up questions

  • How would you adjust your model if the churn definition changed from 'cancellation' to 'inactivity for 60 days'?
  • If you deploy a tree-based model, how would you interpret the feature importance scores while accounting for potential multi-collinearity?
  • How would you approach model calibration to ensure that a predicted probability of 0.2 actually corresponds to a 20% churn rate in practice?