Katkat
Back

Design and validate an ads feed experiment

MetaData Scientist · Onsite
HardData ScientistOnsiteAnalytics & Experimentation

Last updated

This question tests the ability to design a rigorous A/B test for a core product feature, covering metric selection, interference handling, and post-experiment analysis under complex constraints.

The Scenario

Meta is considering introducing a 'Sponsored Video Reel' into the main feed. The goal is to increase ad revenue (RPM - Revenue per Mille) without significantly degrading user engagement (Time Spent). As a Data Scientist, you are tasked with designing the experiment.

Your Task

Part 1: Design

  1. Define the primary metric, a guardrail metric, and a secondary metric. Justify your choices.
  2. How would you determine the necessary sample size if you expect the new ad format to potentially increase revenue by 2% while causing a slight decrease in user sessions?
  3. Would you use user-level, cluster-level, or switchback randomization? Explain the trade-offs regarding interference (network effects).

Part 2: Analysis

  1. You notice a significant increase in revenue, but 'Time Spent' shows a decline that is borderline statistically significant (p = 0.048). How do you handle the multiple testing problem if you are monitoring 10 different metrics?
  2. After the experiment, you find that the treatment group experienced a 'novelty effect' where initial engagement was high but dropped after 14 days. How would you adjust your analysis to predict long-term impact?

Part 3: Validation

  1. How would you check for cross-experiment contamination if other teams are running experiments simultaneously? Provide a method to validate the randomization quality.

Hints

Hint 1

Consider whether the 'Stable Unit Treatment Value Assumption' (SUTVA) holds for a social network where user interactions are interdependent.

Hint 2

Think about the difference between 'confirmatory' and 'exploratory' analysis when deciding how to handle the family-wise error rate for your 10 metrics.

Hint 3

For the novelty effect, consider a time-series decomposition or a segmented analysis comparing cohorts based on their join date or activity level.

Follow-up questions

  • If we implement CUPED (Controlled-experiment Using Pre-Experiment Data) to reduce variance, what assumptions must hold for the pre-experiment covariate?
  • How would you approach a situation where the experiment is interrupted by an external event, such as a major holiday or a global news event?
  • If we see a positive impact on revenue but a negative impact on long-term retention (measured at 30 days), how would you frame the trade-off to the Product Manager?