Katkat
Back

Predict Next-Week Streaming Engagement at Event Scale

SpotifyMachine Learning Engineer · Onsite
HardMachine Learning EngineerOnsiteML System Design

Last updated

This question tests the ability to design an end-to-end ML system for user retention prediction, focusing on feature engineering, model scalability for massive user bases, and real-time inference latency.

Problem Statement

Design a scalable ML system to predict the probability that a Spotify user will be active (defined as at least one stream) in the upcoming week. The system will support personalized retention campaigns.

Functional Requirements

  • Generate a weekly "churn risk" score for all active users.
  • Provide an API for internal marketing teams to query a user's risk score in real-time.
  • Enable cold-start predictions for new users.

Non-Functional Requirements

  • Scale: Support 500 million+ active monthly users.
  • Throughput: Batch inference for the entire user base once per week; real-time inference for individual users upon profile updates.
  • Latency: Real-time queries should return in under 100ms.

Deliverables

Your design should cover:

  1. Data Pipeline: How to handle streaming telemetry data and store features for both training and inference.
  2. Model Architecture: Choices for the model type, handling temporal dependencies, and managing the "active next week" label definition.
  3. Serving Infrastructure: Architecture for batch scoring vs. online serving.
  4. System Robustness: Strategies for monitoring feature drift and handling model failures.

Constraints & Assumptions

  • You have access to a distributed compute cluster (e.g., Spark) and a low-latency NoSQL store (e.g., Cassandra/DynamoDB).
  • User behavior data is sparse and high-dimensional.

Hints

Hint 1

Consider the difference between a global model for all users and user-segment specific models given the massive scale.

Hint 2

Think about how to balance the need for short-term session features (real-time) versus long-term historical trends (batch).

Hint 3

How would you implement a feedback loop to validate the model's predictions if a user is targeted with a retention campaign based on your score?

Follow-up questions

  • How would you handle concept drift if a new, highly viral album changes the definition of 'normal' streaming behavior across the platform?
  • If the marketing team decides to change the target intervention (e.g., from an email to a push notification), how does that impact your model's offline evaluation strategy?
  • What strategies would you use to prune features or perform model distillation if the latency requirements become significantly more stringent?