Katkat
Back

Decide event notification launch via experiments

MetaData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests the ability to design a robust A/B test in the presence of network effects, interference, and cannibalization within a social notification ecosystem.

Scenario

Meta is considering a new notification feature: "Event Social Proof," which informs a user when their close friends have RSVP'd to a public event. You need to design an experiment to decide whether to launch this feature globally.

Your Task

Draft a comprehensive experiment design that addresses the following:

  1. Randomization Strategy: Given that users are interconnected and notifying one user might influence a friend's behavior, how do you randomize to minimize spillover bias?
  2. Metric Definition: Define a primary metric for the test. Beyond short-term engagement, what guardrail metrics would you track to account for notification fatigue or cannibalization of existing notification types?
  3. Power and Ramp: How would you determine the necessary sample size, and what is your strategy for ramping up the experiment (e.g., population segmentation)?
  4. Launch Criteria: What specific statistical evidence or business signals would you require to green-light a full global launch, considering the potential long-term negative impacts on platform health?

Constraints

  • Assume a daily active user base of 1 billion.
  • The system must account for "network interference," where the treatment effect on User A depends on whether their friends also received the treatment.
  • You must address the trade-off between the new feature and existing notification surface area.

Hints

Hint 1

Consider using cluster-based randomization (e.g., by geography or social graph communities) instead of individual-level randomization to isolate network effects.

Hint 2

Define a metric that captures 'Notification Yield,' such as the ratio of meaningful interactions to total notifications sent, rather than just raw click-through rates.

Hint 3

Think about negative control metrics like 'Notification Unsubscribe Rate' or 'Time Spent on Platform' to measure potential cannibalization and user frustration.

Follow-up questions

  • How would you measure the 'network effect' benefit—do users attend more events when they see their friends' activity, or are they just clicking more often?
  • If the experiment shows a positive impact on event RSVPs but a decrease in other types of notifications, how would you calculate the net utility to the user?
  • What would you do if the experiment results look significant, but the effect size is only positive for a small subset of the population (e.g., power users)?