Katkat
Back

Design A/B Test for New Amazon Recommendation Module

AmazonData Scientist · Onsite
HardData ScientistOnsiteAnalytics & Experimentation

Last updated

Design A/B Test for New Amazon Recommendation Module evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Scenario

Amazon wants to test a brand-new product recommendation on the home page.

Question

Design an A/B test to evaluate the new recommendation module. Define test and control, duration and required sample size. What primary metric would you track? How would you justify its business relevance? Define a p-value and explain how it is used to decide whether the experiment is successful. Describe potential sources of bias in this experiment and how you would guard against them. If you can only expose 5 % of users, how would you ensure adequate statistical power? Suppose the treatment lifts click-through-rate but reduces average order value; how would you decide whether to launch? Outline a causal inference approach (e.g., difference-in-differences or propensity matching) you could apply if randomization were impossible.

Hints

Cover hypothesis formulation, metric hierarchy, variance reduction, sequential testing and guardrail metrics.