Katkat
Back

Design A/B Test for Streaming Feature Network Effects

NetflixData Scientist · Other
HardData ScientistOtherAnalytics & Experimentation

Last updated

This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Design A/B Test for Streaming Feature Network Effects states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Scenario

Designing and evaluating an A/B test for a new streaming feature that may create social-network spill-overs.

Question

How would you choose primary and guardrail metrics for this experiment? What threats do network effects pose and how would you address them? Explain SUTVA and why it matters here. Describe how you would run and interpret a multivariate test on title artwork. How would you measure heterogeneous treatment effects across user segments?

Hints

Cover metric definition, experiment unit, variance reduction, and validity assumptions.