Katkat
Back

Interpret a Streaming Regression Without Overstating Causality

HuluData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This problem tests your ability to interpret multivariate regression results for product stakeholders, specifically focusing on distinguishing correlation from causation and identifying risks like collider bias.

The Scenario

You are a Data Scientist at a streaming service. You have built a linear regression model to predict user_retention_score (0-100) based on daily_minutes_watched and a binary feature received_push_nudge (1 if the user received a personalized recommendation nudge, 0 otherwise).

Your model output is as follows:

  • Intercept: 45.2
  • Coefficient for daily_minutes_watched: 0.85 (p < 0.001, 95% CI: [0.82, 0.88])
  • Coefficient for received_push_nudge: 12.4 (p < 0.001, 95% CI: [10.1, 14.7])

Your Product Manager looks at this and says: "Great, sending push nudges increases retention by 12 points. Let's send them to every user to boost our score."

Your Task

  1. Explain to the Product Manager why the 12.4 coefficient does not necessarily imply that the push nudge causes a 12-point increase in retention.
  2. Identify at least one potential confounding variable or selection bias mechanism that could lead to this result.
  3. Propose a more robust experimental design to isolate the true impact of the nudge.

Constraints & Considerations

  • Avoid technical jargon where possible, but be precise in your statistical reasoning.
  • Consider that the marketing team only sends nudges to users who have been inactive for more than 48 hours.

Hints

Hint 1

Consider the 'selection bias' inherent in the criteria used to trigger a nudge.

Hint 2

Think about whether the 'retention score' might have been lower for the treatment group even before the nudge was sent.

Hint 3

Compare the current observational regression with a randomized controlled trial (RCT) framework.

Follow-up questions

  • How would you adjust your model if you found that the nudges were primarily sent to users with high historical engagement who were 'at risk' of churning?
  • What is the risk of 'post-treatment bias' if we included a variable like 'number of clicks on the nudge' in our regression?
  • If an A/B test shows a lower lift than the observational model, how would you explain this discrepancy to the stakeholders?