Katkat
Back

Build a Churn Prediction Model

GustoData Scientist · Onsite
MediumData ScientistOnsiteMachine Learning

Last updated

This question tests the ability to design an end-to-end churn prediction pipeline, focusing on temporal data integrity, class imbalance, and actionable evaluation metrics for business stakeholders.

Scenario

Gusto provides payroll and benefits software to small businesses. You are tasked with building a churn prediction model to identify customers at risk of canceling their subscription within the next 90 days. The business objective is to provide the Customer Success team with a prioritized list of at-risk accounts for proactive outreach.

Data Available

You have access to the following logs:

  1. subscriptions: account_id, start_date, end_date, plan_type.
  2. activity_logs: account_id, timestamp, event_type (e.g., login, payroll_run, tax_filing).
  3. support_tickets: account_id, timestamp, sentiment_score.
  4. billing: account_id, amount_due, payment_status.

Your Task

  1. Label Definition: How do you define a 'churned' customer given that cancellations can be voluntary or involuntary, and some customers have seasonal payroll needs?
  2. Feature Engineering: Describe the process of creating 'point-in-time' features. How do you ensure no data leakage occurs when creating features based on the activity_logs or support_tickets?
  3. Modeling Strategy: The dataset is highly imbalanced (most customers stay). Discuss your approach to handling this imbalance, including your choice of model and loss function.
  4. Evaluation: Why might accuracy be a poor metric here? Which metric would you optimize for, and how would you present the results to the Customer Success team?
  5. Deployment: Given that the business wants to reach out to customers before they cancel, how frequently should this model generate scores, and what level of explainability is required for the support team?

Hints

Hint 1

Consider defining churn as a windowed event (e.g., no payroll run for 45+ days) rather than just a subscription cancellation flag to capture 'silent' churn.

Hint 2

For point-in-time features, imagine you are standing at date 'T'. Any activity after 'T' must be hidden from the training row labeled for time 'T'.

Hint 3

Think about the difference between predicting churn probability and ranking users; consider metrics like Precision-Recall AUC or Lift charts to evaluate how effectively the model identifies the highest-risk group.

Follow-up questions

  • If your model identifies a 'high-risk' account, how would you use SHAP or LIME values to justify the intervention to an account manager?
  • How would you measure the success of the model once the Customer Success team begins acting on the predictions?
  • How would you handle cold-start issues for new customers who have less than 30 days of platform history?