Katkat
Back

Design a clustered A/B test with spillovers

MetaData Scientist · HR Screen
HardData ScientistHR ScreenAnalytics & Experimentation

Last updated

This question tests the ability to design a cluster-randomized experiment when interactions between users (network interference) violate the stable unit treatment value assumption (SUTVA).

The Scenario

You are a data scientist at a social media platform. You want to test a new "viral" notification feature that encourages users to invite friends to join a group. You suspect that if a user in a group receives the notification, their activity might positively influence their friends, even if those friends were not in the treatment group. This spillover effect means that individual-level randomization will lead to biased treatment effect estimates.

The Task

Design an experiment to estimate the causal impact of this notification feature on group-level engagement. Address the following components:

  1. Randomization Strategy: Describe how you would cluster users to minimize interference while maintaining statistical power. How do you define the "cluster" boundary?
  2. Analysis Framework: Propose an estimator for the Average Treatment Effect (ATE). How would you account for intracluster correlation in your variance estimation?
  3. Validation: How would you verify that your randomization successfully mitigated spillover effects during the post-hoc analysis?

Constraints & Considerations

  • Assume the platform has 10 million active users distributed across 500,000 existing groups.
  • You have access to historical user activity data and a friendship graph.
  • Acknowledge the trade-off between bias (due to interference) and variance (due to clustering).

Hints

Hint 1

Consider forming clusters based on the connected components of the user friendship graph rather than arbitrary geographic or temporal segments.

Hint 2

Think about using a cluster-robust standard error estimator to handle correlated outcomes within the same cluster.

Hint 3

You can test for the presence of spillovers by comparing the treatment effect on groups with high vs. low internal connectivity.

Follow-up questions

  • How would your approach change if the spillover effect is not just within groups, but extends to the entire network graph?
  • If your experiment shows a significant effect, how would you distinguish between the 'direct effect' on treated individuals and the 'indirect effect' (spillover) on their connections?
  • How does the choice of cluster size impact your Minimum Detectable Effect (MDE) compared to an individual-level randomized design?