Katkat
Back

Walk through an A/B test end-to-end

AmazonData Scientist · Technical Screen
EasyData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question evaluates a data scientist's competencies in experimental design, statistical inference, causal reasoning, metric selection, and data quality assurance for A/B testing.

Walk through how you would design, run, and analyze an A/B test for a product change.

Your answer should include:

  • Hypothesis framing and choosing primary, diagnostic, and guardrail metrics.
  • Experiment design: unit of randomization, population, exposure definition, duration, and handling novelty/seasonality.
  • How you determine sample size / MDE / power.
  • Data quality checks (e.g., SRM), logging issues, and how you validate randomization.
  • Statistical analysis approach (confidence intervals, p-values, multiple testing, sequential peeking).
  • How you interpret results and make a launch decision, including practical vs statistical significance.
  • Common pitfalls (e.g., interference/network effects, noncompliance, missing data).