Test a Post-Creation Tab While Accounting for User Interference
QuoraData Scientist · Technical ScreenLast updated
Design an experiment for a post-creation tab with creator–reader spillovers, cluster-aware analysis and an evidence-based launch decision.
Test a Post-Creation Tab While Accounting for User Interference
A question-and-answer platform is considering adding a post-creation tab next to its existing question tab. Design a test to determine whether this is a good idea, from defining the hypothesis through analysis and a launch recommendation. Address interference: content created by treated users may be read or interacted with by control users, so one user's treatment can affect another user's outcome.
Clarify what the new post format allows and what success means before choosing an outcome. No traffic volume, network structure, experiment duration or effect size is supplied. Explain how you would select a randomization unit and comparison strategy, estimate the necessary sample or duration, and interpret the results when spillovers cannot be eliminated completely.
Follow-up questions
- How could an individual-level experiment underestimate or overestimate the effect of a platform-wide rollout?
- What tradeoffs arise when clustering users reduces spillovers but also reduces the number of independent experimental units?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.