Katkat
Back

Design an Experiment to Evaluate New Recommendation Model

MetaData Scientist · Onsite
HardData ScientistOnsiteAnalytics & Experimentation

Last updated

This problem tests the ability to design an end-to-end A/B test framework for a machine learning model update, focusing on metric selection, causal validity, and post-experiment analysis.

Scenario

You are a Data Scientist at a social media company. The engineering team has developed a new recommendation model for the 'Feed' section that aims to increase long-term user engagement. You need to design an experiment to evaluate whether this model should be launched globally.

Requirements

  1. Metric Definition: Define both the primary evaluation metric and guardrail metrics to ensure the model does not negatively impact the ecosystem.
  2. Experiment Design: Describe the randomization unit and how you would handle potential network interference (spillover effects) between treatment and control groups.
  3. Analysis Plan: Explain the statistical approach to determine significance, specifically how you would account for novelty effects (where users click more simply because the interface changed).
  4. Validation: Propose a method to verify if the experiment results are internally consistent before making a go/no-go decision.

Constraints

  • Assume the platform has 100 million daily active users.
  • Assume your model influences both 'Time Spent' (a proxy for engagement) and 'Ad Revenue' (a core business metric).
  • Consider the trade-off between short-term gains and long-term user retention.

Hints

Hint 1

Consider using Ratio metrics for engagement to normalize for differences in user activity levels across buckets.

Hint 2

Think about how user-level randomization might be insufficient if users interact heavily with shared content, and explore cluster or geographic randomization as alternatives.

Hint 3

To address novelty effects, analyze the treatment effect trend over time to see if the uplift decays as the new model becomes 'familiar' to the user.

Follow-up questions

  • How would you adjust your analysis if the treatment effect was positive for power users but negative for casual users?
  • If your primary metric shows a significant lift but guardrail metrics like 'report rate' or 'block rate' also increase, how do you decide whether to launch?
  • How would you design a 'holdback' experiment to measure the long-term cumulative impact of the recommendation model on churn?