Katkat
Back

Analyze an A/B test over last 7 days

AmazonData Scientist · Onsite
HardData ScientistOnsiteAnalytics & Experimentation

Last updated

This question evaluates competency in A/B test analysis and experiment design, covering conversion-rate computation, statistical inference (two-proportion testing and confidence intervals), sample-ratio checks, heterogeneity assessment (fixed versus random effects), and power/MDE reasoning.

Assume today is 2025-09-01. You ran a 50/50 A/B test on a checkout flow over the last 7 days (2025-08-26 to 2025-09-01). Daily exposures and purchases are below: Date | A_exposed | A_purchases | B_exposed | B_purchases -----------+-----------+-------------+-----------+------------ 2025-08-26 | 28000 | 1350 | 27800 | 1420 2025-08-27 | 28500 | 1380 | 28100 | 1460 2025-08-28 | 28200 | 1390 | 27900 | 1450 2025-08-29 | 28400 | 1420 | 28000 | 1520 2025-08-30 | 28300 | 1370 | 27700 | 1480 2025-08-31 | 28100 | 1410 | 27900 | 1510 2025-09-01 | 30500 | 2580 | 30000 | 2560 Tasks:

  1. Compute overall conversion rates for A and B, absolute/relative lift, and a two-proportion z-test p-value and 95% CI for the lift.
  2. Check for sample ratio mismatch (SRM) daily and overall. If detected, propose root causes and mitigation.
  3. Analyze heterogeneity across days; is it appropriate to pool? Justify with a fixed-effects vs random-effects framing.
  4. Identify at least three pitfalls relevant to this window (e.g., seasonality/holiday effect on 2025-09-01, novelty effects, user overlap, peeking). Propose guardrails you would set before launching.
  5. If your minimal detectable effect (MDE) was a +5% relative lift over baseline, assess achieved power approximately and whether you would roll out, iterate, or extend the test. State any assumptions.