Katkat
Back

Design and Monitor a Multi-Country Parameter Experiment

HuluData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests the ability to design an A/B test across multiple regions while accounting for sample size requirements, variance reduction techniques, and sequential monitoring protocols.

Scenario

Streaming platform 'StreamFlow' is testing a new content recommendation algorithm across three key markets: the US, UK, and Japan. You are tasked with designing an experiment to measure the impact on 'Total Stream Time per User'.

Objectives

  1. Power Analysis: The baseline daily stream time is 120 minutes with a standard deviation of 40 minutes. You want to detect a minimum detectable effect (MDE) of 3 minutes. Calculate the required sample size per variant per country to achieve 80% power at a 5% significance level.
  2. Variance Reduction: Given that historical stream time data is highly skewed and noisy, propose one statistical technique to reduce the variance of your estimator.
  3. Sequential Monitoring: Due to engineering constraints, the product team wants to peek at the results every 48 hours to decide whether to stop early. Explain the risk of this approach and how you would mitigate it.
  4. Cross-Country Aggregation: How do you handle cases where the treatment effect is statistically significant in the US but neutral in Japan? Should you roll out the feature globally?

Constraints

  • Assume 50/50 split.
  • Ignore population size caps; assume sufficient traffic exists.
  • Explain your logic for the cross-country aggregation decision, focusing on the trade-off between global optimization and local user experience.

Hints

Hint 1

Use the sample size formula for a two-sample t-test: n = 16 * variance / (MDE^2).

Hint 2

Consider using CUPED (Controlled-experiment Using Pre-Experiment Data) to normalize metrics based on user history.

Hint 3

Look into Alpha-spending functions or sequential probability ratio tests (SPRT) to manage the Type I error rate inflation caused by multiple peeking.

Follow-up questions

  • How would you detect and account for interference (e.g., social network effects) if the treatment in one country influences users in another?
  • What specific guardrail metrics would you implement to ensure that increased stream time doesn't lead to higher subscriber churn?
  • If the algorithm is computationally expensive to run, how would you re-design the experiment to use a smaller sample while maintaining statistical power?