Design A/B Test to Evaluate Algorithm's Revenue Impact

Scenario

You are responsible for evaluating the lift of a new recommendation algorithm via online experiments.

Question

Design an A/B test to measure the algorithm’s impact on revenue: define hypotheses, choose unit of randomization, compute required sample size, and detail success metrics. After the experiment you obtain p = 0.08 for revenue lift; interpret this result and recommend whether to ship. Explain how you would estimate causal impact if randomization were not possible; compare methods such as difference-in-differences, propensity score matching, and instrumental variables.

Hints

Demonstrate knowledge of hypothesis testing, power analysis, Type I/II errors, and causal inference techniques.