Connect Probability, Experiment Sizing, and Bootstrap Inference

Scenario

You are the lead data scientist evaluating a new ranking algorithm for the Pinterest Home Feed. The primary engagement metric is repins_per_user over a 14-day experiment window, accompanied by the primary conversion metric user_save_rate (proportion of users who save at least one pin).

Answer the following four linked technical questions:

1. Discrete Probability & Session Dynamics

Assume each session consists of a sequence of independent pin impressions. In any given session, a user views pins one by one until they save a pin, after which the session terminates. For each pin, the baseline probability of a save is p = 0.08.

• What is the probability that a user views more than 10 pins in a single session?
• Suppose a user has already viewed 8 pins without saving. What is the conditional probability that they view more than 12 pins total?

2. Experiment Sizing

You need to power a two-tailed, two-sample test on user_save_rate with baseline proportion p0 = 0.25. You want to detect a minimum relative lift of 2% (p1 = 0.255) with statistical power of 80% (beta = 0.20) at significance level alpha = 0.05.

• Write the explicit formula for sample size n per variant using the normal approximation to the binomial distribution.
• Explain how the required sample size changes if you test a continuous metric like repins_per_user where the variance is 4 times the mean (sigma^2 = 4 * mu), compared to a Poisson distribution assumption.

3. Bootstrap Inference on Heavy-Tailed Metrics

At the end of the experiment (with N = 100,000 users per variant), you analyze repins_per_user. Over 70% of users have 0 repins, while the top 1% of users account for 45% of total repins.

• A standard two-sample t-test yields p = 0.038 in favor of the treatment.
• You run 10,000 bootstrap resamples to compute the 95% percentile bootstrap confidence interval for the difference in means (mu_treatment - mu_control), obtaining [-0.004, 0.052].
• Explain the mathematical reason behind the discrepancy between the asymptotic t-test and the percentile bootstrap CI. Which method is more trustworthy here, and why?

4. Product Decision

The product manager wants to ship the feature immediately because the t-test was "statistically significant at p < 0.05". Outline your recommendation, how you would address the PM's argument, and what additional checks (e.g., metric transformation, outlier trimming, or stratified bootstrap) you would run before making a final call.