Plan and analyze an A/B test

The Problem

Amazon is launching a new 'One-Click Prime' checkout feature. You are tasked with designing an A/B test to measure the impact on the conversion rate, currently 10%. You aim to detect a minimum detectable effect (MDE) of 0.5% (relative lift) with 80% power at a significance level (alpha) of 0.05.

However, there are two constraints:

1. Clustered Variance: Users belong to households. You observe an intra-class correlation coefficient (ICC) of 0.05. The average household size is 2.5 users. You must account for the design effect in your sample size calculation.
2. Sequential Monitoring: The team wants to check results daily for 10 days to potentially stop the test early if a significant result is found. You must maintain the overall family-wise error rate at 0.05.

Task

1. Calculate the required sample size ($N$) for a standard A/B test (ignoring clusters and sequential testing).
2. Adjust $N$ to account for the design effect due to household clustering.
3. Briefly explain how you would adjust the p-value threshold for the sequential monitoring to ensure the 0.05 alpha is preserved.

Constraints & Definitions

• Use the standard normal approximation for power calculations: $N = [Z_{alpha/2} + Z_{beta}]^2 * [p(1-p) * 2] / [delta^2]$.
• Design Effect ($DE$) = $1 + (m - 1) * ICC$, where $m$ is the average cluster size.
• Assume $Z_{0.025} = 1.96$ and $Z_{0.20} = 0.84$.

Example

If the variance of the mean of a cluster is inflated, the effective sample size is reduced. For a total of 10,000 users, if the design effect is 1.07, the effective sample size is approximately 10,000 / 1.07.