Katkat
Back

Design and analyze pricing-page A/B test

AmazonData Scientist · Onsite
HardData ScientistOnsiteAnalytics & Experimentation

Last updated

This question evaluates data-science competencies in online experiment design and analysis, including statistical power and sample-size estimation, variance-reduction methods, guardrail and SRM monitoring, randomization and contamination handling, sequential monitoring and stopping rules, heterogeneity/subgroup analysis, and result communication within the analytics & experimentation domain. It is commonly asked to assess practical application of statistical inference and experimental rigor in product and business contexts, primarily testing practical application while requiring conceptual understanding of underlying statistical principles and operational trade-offs.

You plan to test a new pricing-page layout. Primary metric: Paid conversion within 7 days of first page view at the user level. Baseline conversion = 6.0%; target MDE = +5% relative (to 6.3%); α = 0.05 (two-sided); power = 0.80. Daily traffic: 200,000 unique users; 60% new, 40% returning; 10% tablet, 30% desktop, 60% mobile. Constraints: users may switch devices; bots must be filtered; some users arrive via paid ads with different intent. Tasks: (1) Compute the approximate per-variant sample size and minimum test duration with and without CUPED (assume CUPED yields 20% variance reduction); (2) Define guardrails (e.g., SRM, bounce rate, revenue/user) and how you will detect SRM in near real time; (3) Specify randomization unit (cookie vs user vs account), handling cross-device identity, and how to prevent contamination; (4) Plan for sequential looks while controlling Type I error (e.g., alpha-spending) and describe a pre-registered stopping rule; (5) Outline a heterogeneity analysis across device and traffic source, and how you would proceed if the average treatment effect is neutral but mobile shows a significant lift; (6) Provide a concrete plan for analyzing and communicating results, including effect sizes with 95% CIs and a decision framework for ship/iterate/halt.