Katkat
Back

Solve core probability and statistics questions

NetflixData Scientist · Onsite
EasyData ScientistOnsiteStatistics & Math

Last updated

This problem tests your understanding of the Central Limit Theorem and its application in power analysis for A/B testing and confidence interval estimation.

The Problem

Netflix is testing a new recommendation algorithm. You have historical data showing that user session times are distributed with a mean of 45 minutes and a standard deviation of 15 minutes. The distribution is skewed, not normal.

  1. If you randomly sample 100 users, what is the approximate distribution of the sample mean session time? Why does this hold even though the underlying population distribution is skewed?

  2. Suppose you want to detect a minimum detectable effect (MDE) of 2 minutes in the mean session time using a two-tailed test with a significance level (alpha) of 0.05 and a power of 0.80. Given a population standard deviation of 15, calculate the required sample size per group. Assume the standard normal Z-score for alpha/2 (0.025) is 1.96 and for beta (0.20) is 0.84.

  3. Explain how the width of the confidence interval for the sample mean changes if the sample size increases by a factor of 4. Support your answer with the mathematical formula for the confidence interval.

Constraints

  • Use the standard normal approximation for power calculations.
  • Assume users are independent and identically distributed.

Formula for Sample Size

n = 2 * (Z_alpha/2 + Z_beta)^2 * sigma^2 / delta^2 where delta is the MDE.

Hints

Hint 1

Recall the Central Limit Theorem: the sum (or mean) of a large number of independent variables tends toward a normal distribution regardless of the initial distribution.

Hint 2

For the sample size calculation, plug the given values into the provided formula: Z_alpha/2 = 1.96, Z_beta = 0.84, sigma = 15, and delta = 2.

Hint 3

Observe the relationship between the sample size (n) and the margin of error (standard error term) in the confidence interval formula `mean +/- Z * (sigma / sqrt(n))`.

Follow-up questions

  • How would you adjust your power analysis if you discovered that session times are highly correlated within certain geographic regions?
  • If the underlying population distribution was known to be very heavy-tailed, would the Central Limit Theorem still be a reliable tool for your inference? Why or why not?
  • How does the choice of alpha (significance level) trade off with the required sample size and the risk of Type II errors?