Katkat
Back

Compute minimum sample size for A/B test

RobloxData Scientist · Online Assessment
HardData ScientistOnline AssessmentAnalytics & Experimentation

Last updated

Evaluates the ability to perform sample size and power calculations for a two-sample z-test—including estimating outcome standard deviation from historical continuous data—and is categorized under Analytics & Experimentation for a Data Scientist role at an implementation-level applied statistics abstraction.

You are implementing a function to compute the minimum total sample size for an A/B test.

You are given:

  • observed: a 1D array of historical/baseline metric values (continuous outcome) to estimate the outcome standard deviation.
  • alpha: significance level for a two-sided test (e.g., 0.05).
  • power: desired statistical power (e.g., 0.8).
  • delta: the minimum detectable absolute change in the mean (treatment mean − control mean) you want to be able to detect.

Assumptions:

  • Two-sample z-test for difference in means.
  • Treatment and control groups are equal-sized.
  • Outcome variance is the same in both groups and is estimated from observed.
  • Use the normal approximation (z critical values).

Task:

  1. Estimate (\sigma) using the sample standard deviation of observed.
  2. Compute the minimum required per-group sample size (n).
  3. Return the minimum total sample size (N=2n) as an integer, rounded up to the next integer if needed.

Clearly state the formula you use and any edge-case handling (e.g., delta <= 0, sigma == 0).