Katkat
Back

Handle multiple experiment metrics and communication

TubiData Scientist · Technical Screen
HardData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question assesses the ability to design a multi-metric experiment, manage the false positive rate under multiple testing, and communicate trade-off decisions to non-technical stakeholders.

Scenario

Tubi is testing a new content recommendation algorithm for the 'Recommended for You' carousel. The primary metric is 'Average Watch Time per Session' (primary). You have three guardrail metrics: 'App Crash Rate,' 'Ad Load Latency,' and 'Search Query Frequency.' Additionally, you have two secondary metrics: 'Click-Through Rate (CTR)' and 'Content Diversity Score.'

Task

  1. Propose an experimental design, including power calculation considerations for these heterogeneous metrics.
  2. Explain how you would manage the risk of False Positives given the multiple metrics being tracked.
  3. Assume the primary metric (Watch Time) increases by 2%, but 'Ad Load Latency' increases by 5% (p < 0.05). Draft a brief email (max 150 words) to the Product Manager explaining whether to launch the feature, highlighting the trade-offs.

Constraints & Considerations

  • Assume a 2-week experiment duration.
  • Define how you handle the dependency between 'Watch Time' and 'Ad Load Latency'.
  • Describe the criteria you would set for 'pre-registration' to prevent p-hacking.

Hints

Hint 1

Consider using the Bonferroni correction or Benjamini-Hochberg procedure to adjust your significance thresholds.

Hint 2

Think about whether the guardrail metric impact is a 'hard stop' or a 'business negotiation' point based on its magnitude.

Hint 3

Structure your communication by stating the recommendation clearly first, followed by the supporting evidence and the identified risks.

Follow-up questions

  • How would your analysis change if the experiment shows a significant result for the primary metric but a counter-intuitive drop in a secondary metric?
  • What strategies would you use if the 'Ad Load Latency' is found to be non-stationary (e.g., changes over time during the experiment)?
  • How do you handle the interpretation of results if you observe Simpson's Paradox across different user segments (e.g., power users vs. new users)?