Katkat
Back

Evaluate Notification-Based Account Ranking

MetaData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests the ability to design an experiment to measure the causal impact of a new notification strategy on account ranking quality and user engagement.

Scenario

Meta is considering a new ranking algorithm for the 'Accounts You May Know' (AYMK) feature. Currently, the ranking relies on social graph proximity. The team wants to test if sending 're-engagement' push notifications to users based on predicted interest—rather than just social connections—improves long-term account discovery.

Task

Design an experiment to evaluate the impact of this new notification-based ranking on user retention and account discovery. Address the following:

  1. Metric Definition: Define a primary metric to capture the success of the new ranking strategy and at least one 'guardrail' metric to monitor for negative side effects.
  2. Experiment Design: How would you randomize the experiment? What are the potential risks of cannibalization or network effects in this context?
  3. Causal Inference: How would you account for users who have disabled push notifications or those who rarely open the app?
  4. Statistical Significance: If your primary metric shows a 0.5% lift, how would you determine if this is practically significant versus statistically significant given the scale of the user base?

Constraints

  • Assume the platform has 2 billion daily active users.
  • Consider that push notification delivery is not instantaneous and relies on client-side state.
  • You must identify at least two potential confounding variables that could bias your results.

Hints

Hint 1

Consider the difference between 'intent-based' clicks and 'accidental' notification clicks when defining your primary metric.

Hint 2

Network effects often lead to interference between treatment and control groups; think about how 'cluster randomization' could mitigate this.

Hint 3

Guardrail metrics should focus on user trust and churn, such as notification opt-out rates or report-as-spam triggers.

Follow-up questions

  • How would you measure the long-term cumulative effect of these notifications, given that users might develop 'notification fatigue' over time?
  • If the experiment shows a positive lift in the first week but a flat result after one month, what hypotheses would you test to explain the decay?
  • How would you adjust your analysis if the treatment group shows higher engagement but also a significant increase in app uninstalls?