Katkat
Back

Design experiment on culture memo emphasis

NetflixData Scientist · HR Screen
MediumData ScientistHR ScreenAnalytics & Experimentation

Last updated

This question tests the ability to design a rigorous A/B test for a qualitative product change, focusing on metric selection, potential bias mitigation, and pre-experimental planning.

Scenario

Netflix is considering a change to the onboarding flow where users are presented with a 'Culture Memo' summary card that emphasizes the company's core values before they select their first profile. The Product team believes this builds brand affinity and improves long-term retention. As a Data Scientist, design an experiment to evaluate the impact of this feature.

Requirements

  1. Hypothesis and Metrics: Clearly state your primary metric for success and define at least two guardrail metrics to monitor for negative side effects.
  2. Experimental Design: Describe your randomization strategy. What is the unit of randomization, and how will you ensure the experiment is clean?
  3. Power and Sensitivity: Briefly explain how you would determine the necessary sample size. If the conversion rate for the primary metric is low, what techniques would you use to improve the sensitivity of your test?
  4. Validity: How would you check for Sample Ratio Mismatch (SRM) and identify potential novelty effects versus long-term behavioral changes?

Provide your design in the form of an experimental plan that a PM could follow.

Hints

Hint 1

Consider that the 'Culture Memo' might cause users to abandon the signup flow, so define guardrail metrics that track funnel progression.

Hint 2

Think about whether user-level randomization or session-level randomization is more appropriate for a one-time onboarding experience.

Hint 3

To improve sensitivity, consider using CUPED (Controlled-experiment Using Pre-Experiment Data) to reduce variance by adjusting for pre-treatment user behavior.

Follow-up questions

  • If we find that the new memo increases retention for new users but decreases signup completion rate, how do we decide whether to roll it out?
  • How would you account for seasonality if you run this experiment during a major content release window?
  • What specific segments would you analyze to check for heterogeneous treatment effects?