Evaluate New Feed-Ranking Algorithm with A/B Testing

Scenario

A social-media company wants to evaluate a new feed-ranking algorithm intended to increase daily active minutes.

Question

a) Formulate the A/B test hypothesis (null and alternative) and select primary and guardrail metrics.

b) Determine the minimal detectable effect and required sample size for 95% power, two-tailed α = 0.05.

c) After launch, the product dashboard shows a time-series chart of average active minutes by group. Describe what you look for to confirm experiment health (e.g., parallel pre-period, no data loss) and how you would interpret a sudden mid-test dip.

d) Explain how you would establish causal inference if rollout is geography-based rather than randomized.

Hints

Cover randomization, practical significance, and difference-in-differences when random assignment is impossible.