Katkat
Back

Design Experiments Under Network Interference

PlaystationData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This problem tests your understanding of causal inference in networked environments, focusing on how to detect and mitigate bias when treatment effects spill over between connected users in a social gaming platform.

Scenario

You are a Data Scientist at a gaming company. You want to test a new "Social Quest" feature that allows players to invite friends to complete tasks for bonus rewards. You hypothesize that the feature will increase overall playtime. You plan to conduct an A/B test by randomizing players into two groups. However, you are concerned about network interference: players in the treatment group may invite friends who are in the control group, potentially causing the control group to be exposed to the feature's effects (spillover).

Task

  1. Explain why a standard A/B test (randomizing by individual User ID) will likely produce a biased estimate of the Average Treatment Effect (ATE) in this scenario. In which direction is the bias likely to go?
  2. Propose an alternative randomization strategy to minimize this interference. What are the trade-offs regarding statistical power and variance?
  3. How would you design a diagnostic metric or an empirical check to quantify the level of spillover in your current experimental data?

Example

Imagine a graph where nodes are players and edges are friendship links.

  • Group T: Players with access to the Social Quest.
  • Group C: Players without access.
  • Spillover: A player in T invites a friend in C. The friend in C now participates in the quest, potentially increasing their playtime, even though they were assigned to C.

Constraints

  • Assume the platform has 10 million daily active users.
  • Assume you have access to a full adjacency matrix of friendship links.
  • Your answer should focus on the causal validity of the estimate rather than computational efficiency.

Hints

Hint 1

Consider how the 'Stable Unit Treatment Value Assumption' (SUTVA) is violated when one player's treatment influences another player's outcome.

Hint 2

Think about clustering: instead of randomizing individuals, could you randomize groups of connected players (e.g., connected components or ego-networks)?

Hint 3

For the diagnostic check, consider comparing the outcome of control units with many treated friends against control units with few treated friends.

Follow-up questions

  • How would you adjust your variance estimation if you shift from individual-level randomization to cluster-level randomization?
  • If the network is too dense (e.g., a 'small world' graph where almost everyone is connected), is clustering still a viable strategy? What else could you do?
  • How does the 'interference' problem change if the goal is to measure the total effect of the feature on the entire platform versus the direct effect on individual users?