Katkat

Design an experiment for spam filtering impact

SnapchatData Scientist · Technical Screen

Last updated

This question evaluates experimental design and causal inference skills for measuring the effect of a stricter spam filter on same-day friend-request acceptance, covering hypothesis specification, interference and randomization choices, metric definition and windows, power calculation, and handling incomplete labels.

You plan to launch a stricter spam filter and want to understand its impact on friendship dynamics and same-day acceptance.

a) State clear hypotheses: primary (effect on same-day acceptance rate) and at least two guardrail metrics (e.g., total requests sent, approval latency, false-positive spam rate). Specify null/alternative precisely. b) Propose the experimental unit and randomization scheme that mitigates network interference (e.g., cluster by requester vs. recipient). Justify your choice and discuss spillover risks. c) Define primary and secondary metrics, including exact measurement windows and UTC date boundaries. Explain how to handle delayed approvals that cross days. d) Outline your sample-size/power plan: assumed baseline same-day acceptance, minimum detectable effect, variance source, test duration, and a plan for sequential looks while controlling Type I error. e) Describe how incomplete spam labels in users could bias your metrics and two mitigation strategies (e.g., unknown bucket + sensitivity bounds; propensity/inverse-probability weighting if labeling is MAR). Explain how you would report results with these adjustments.