Design cluster-randomized test under network effects
MetaData Scientist · Technical ScreenLast updated
This question evaluates a data scientist's experiment design and causal inference skills under interference, covering exposure and estimand definition, graph-based cluster construction, cluster-level randomization and contamination mitigation, metric selection, and design-effect/sample-size calculations.
Design an A/B test for a new Group Call feature where network effects and interference are expected. Requirements: (a) Define exposure and the estimand (e.g., ITT at the cluster level vs per-user ATE under interference). (b) Construct clusters from the social graph (describe edge weighting for 'strong ties', the clustering rule—e.g., connected components after thresholding—and how to prevent cluster overlap). (c) Randomize at the cluster level and explain how to handle cross-cluster edges (frontier users), including holdout buffers or partial saturation. (d) Choose a primary metric and two guardrails; critically assess 'time spent per user per day' as a success metric (pros, cons, manipulation risks), and propose at least one alternative primary metric. (e) Compute the design effect and sample-size inflation for cluster randomization: if average cluster size m=20 and intracluster correlation ICC=0.05, what is the design effect and how does it change if m doubles but ICC halves? (f) If you ignore network effects and randomize by user, under what conditions will the naïve estimate be biased downward vs upward? Provide intuition for positive vs negative spillovers and give one real-world case for each direction.
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.