Katkat
Back

Compute A/B test sample size and Bayes posterior

RobloxData Scientist · Online Assessment
EasyData ScientistOnline AssessmentStatistics & Math

Last updated

This question tests the ability to calculate required sample size for a frequentist A/B test using power analysis and to perform a Bayesian update for a Bernoulli process.

Part 1: Sample Size Calculation

Suppose you are designing an A/B test for a feature on a platform. Your current conversion rate is 5% (p = 0.05). You want to detect a relative lift of 10% (i.e., your target conversion rate is 5.5%). Calculate the required sample size per variant to achieve a statistical power of 0.80 and a significance level (alpha) of 0.05.

Use the formula: n = [Z(alpha/2) * sqrt(2 * p * (1-p)) + Z(beta) * sqrt(p*(1-p) + p_alt*(1-p_alt))]^2 / (p_alt - p)^2 where Z(0.025) = 1.96 and Z(0.20) = 0.84.

Part 2: Bayesian Update

Assume you are modeling the conversion probability of a new user segment using a Beta distribution. You start with a prior distribution of Beta(alpha=2, beta=18), representing your belief about the 10% conversion rate. You then observe 100 users, where 8 users converted. Calculate the parameters of the posterior distribution, Beta(alpha', beta').

Example:

If you have a prior Beta(1, 1) and observe 1 conversion out of 10 trials, the posterior is Beta(1+1, 1+9) = Beta(2, 10).

Constraints:

  • Assume independence of observations.
  • Round final sample sizes up to the nearest integer.
  • Provide the updated alpha and beta values for the posterior.

Hints

Hint 1

For Part 1, substitute p=0.05, p_alt=0.055, and the provided Z-scores into the sample size formula.

Hint 2

For Part 2, remember that for a Beta prior and Bernoulli likelihood, the posterior distribution is simply Beta(alpha + successes, beta + failures).

Hint 3

Ensure you distinguish clearly between the frequentist power calculation requirements and the Bayesian parameter conjugation.

Follow-up questions

  • How would the required sample size change if you decided to decrease the significance level to 0.01?
  • In the Bayesian scenario, what is the expected conversion rate (mean) of the posterior distribution?
  • If the variance of your conversion metric is significantly higher than 0.05 * 0.95, how does that impact the validity of the z-test assumption?