Katkat
Back

Evaluate a Live-Stream Group Notification Under Network Effects

MetaData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests the ability to design an experiment that accounts for spillover effects and user fatigue in a social graph, while proposing causal inference methods for environments where randomization is restricted.

The Problem

Meta is testing a new notification feature: when a user goes live, their "Close Friends" group receives a push notification. You are tasked with designing an evaluation framework for this feature.

Direct notification clicks are a noisy metric because they ignore the organic discovery of the live stream by users who saw the notification but opened the app via other means, or those influenced by friends who received the notification.

Your Tasks

  1. Randomization Strategy: Given that users in a group share a social context, explain why simple user-level randomization (A/B testing) might underestimate or overestimate the total treatment effect. Propose an alternative randomization unit that mitigates network interference.

  2. Metric Definition: Beyond click-through rate (CTR), propose two metrics that capture the "network effect" of these notifications. How would you quantify the impact of notification-induced viewers on the total time spent by other users in the stream?

  3. Causal Inference: Suppose you are forbidden from running an A/B test due to engineering constraints. Describe a quasi-experimental approach (e.g., Difference-in-Differences or Synthetic Control) to estimate the impact. How would you define your treatment and control groups, and what identifying assumption must hold for your result to be credible?

  4. Fatigue Mitigation: Propose a method to measure the "wear-out" effect of these notifications over a 30-day period. How do you distinguish between long-term value creation and short-term notification fatigue?

Hints

Hint 1

Consider the 'Stable Unit Treatment Value Assumption' (SUTVA) and how a notification sent to one friend might influence the behavior of another friend who didn't receive it.

Hint 2

Think about cluster-based randomization where you randomize entire social sub-graphs or 'communities' rather than individual users.

Hint 3

For the quasi-experimental approach, look for a staggered rollout (e.g., region-based) to leverage time-series variation.

Follow-up questions

  • If we observe a significant increase in live-stream views but a decrease in overall app engagement, how do you reconcile these conflicting signals?
  • How would you account for 'counterfactual' viewership—would these users have watched the stream anyway if they hadn't received the notification?
  • How does your evaluation strategy change if the notification is sent to a group of 100 people versus a group of 5?