Katkat
Back

Calculate Conversion Lift CI

CoinbaseData Scientist · Onsite
MediumData ScientistOnsiteStatistics & Math

Last updated

This question tests the ability to derive and calculate a confidence interval for the difference in conversion rates between two groups, and to explain how this interval informs a business decision.

Problem Statement

You are analyzing a product experiment conducted on the Coinbase mobile app to see if a new "Instant Buy" button increases user conversion rates compared to the existing checkout flow.

You have the following data from a randomized controlled trial (RCT):

  • Control Group (A): 10,000 visitors, 500 conversions.
  • Treatment Group (B): 10,000 visitors, 560 conversions.

Task

  1. Calculate the point estimate of the conversion lift (the difference in conversion rates between Group B and Group A).
  2. Construct a 95% Confidence Interval (CI) for the difference in conversion rates using the normal approximation method. Assume the critical value z for a 95% confidence level is 1.96.
  3. Based on your interval, state whether the result is statistically significant at the 5% level and explain the practical business implication of your finding.

Requirements

  • Provide the formulas used for the standard error of the difference in proportions.
  • Show your step-by-step calculations.
  • Assume p1 and p2 are the conversion probabilities for the control and treatment groups, respectively.

Example Calculation Format

  • Point Estimate: p2 - p1
  • Standard Error: sqrt(p1(1-p1)/n1 + p2(1-p2)/n2)
  • Interval: (p2 - p1) +/- z * SE

Hints

Hint 1

Recall that the variance of the difference between two independent proportions is the sum of the individual variances of the proportions.

Hint 2

The standard error of a proportion is calculated as `sqrt(p * (1 - p) / n)`, where `p` is the conversion rate and `n` is the sample size.

Hint 3

If the resulting confidence interval for the difference in proportions includes zero, the null hypothesis (that there is no difference) cannot be rejected at that confidence level.

Follow-up questions

  • How would your approach change if the conversion rates were extremely small, e.g., 0.05%?
  • If you were told that the p-value was 0.04, how would you explain the significance of the result to a non-technical Product Manager?
  • What steps would you take if you discovered a significant imbalance in user segments (e.g., high-net-worth vs. retail) between the two groups after the experiment concluded?