Katkat
Back

Define scalable train/validation for churn

HBOData Scientist · Online Assessment
HardData ScientistOnline AssessmentMachine Learning

Last updated

This question assesses the ability to design an end-to-end churn prediction pipeline, focusing on temporal integrity, handling class imbalance, and ensuring robust production deployment.

Scenario: Weekly Subscription Churn Prediction

You are building a churn prediction system for a streaming platform with millions of users. The business defines 'churn' as a user failing to renew their subscription for the upcoming week. You have access to user activity logs, payment history, and content engagement data.

Task

Design a scalable production pipeline that addresses the following constraints:

  1. Temporal Integrity: How do you ensure no 'data leakage' when training on historical user behavior to predict next-week churn?
  2. Class Imbalance: Churners typically represent <5% of the user base. Propose a strategy for handling this imbalance during training and evaluation.
  3. Evaluation Strategy: Given the weekly nature of the target, define a validation strategy that mimics the production deployment lifecycle.
  4. Operationalization: Explain how you would implement model monitoring to detect feature drift and concept drift once the model is live.

Requirements

  • Provide a high-level architecture diagram (described in text).
  • Justify your choice of evaluation metrics (e.g., AUC-ROC vs. Precision-Recall AUC vs. Cost-weighted F1).
  • Describe the data processing workflow for generating weekly snapshots of features for millions of users.

Hints

Hint 1

Consider using 'time-series cross-validation' or a sliding window approach rather than simple random k-fold splits to prevent temporal leakage.

Hint 2

For evaluation, prioritize metrics that are invariant to class distribution, such as PR-AUC or a custom utility function that penalizes false negatives more heavily than false positives.

Hint 3

For production monitoring, compare the distribution of incoming inference features against training distributions using statistical tests like Kolmogorov-Smirnov.

Follow-up questions

  • How would you handle cold-start users who have less than one week of activity history?
  • If the model inference needs to run in under 200ms per user, how would you optimize the feature retrieval layer?
  • How would you calibrate the model output probabilities to ensure they reflect true churn risk, especially if you used downsampling for class imbalance?