Katkat
Back

Derive variance and CTR confidence intervals

RobloxData Scientist · Technical Screen
MediumData ScientistTechnical ScreenStatistics & Math

Last updated

This problem tests your understanding of statistical inference for binomial proportions, specifically calculating confidence intervals for Click-Through Rates (CTR) and understanding how to aggregate variability across multiple days.

The Scenario

You are analyzing the daily Click-Through Rate (CTR) for a new social feature. Over a period of 7 days, you have tracked the following data for each day $i$:

  • $n_i$: Total number of impressions.
  • $k_i$: Total number of clicks.
  • $p_i = k_i / n_i$: The daily CTR.

Tasks

  1. Point Estimation: Given the 7-day dataset, derive the pooled estimator for the overall CTR ($_p$) and the variance of this estimator.
  2. Confidence Intervals: Explain the fundamental difference between the Wald interval (using the normal approximation) and the Clopper-Pearson interval. When might the Wald interval fail?
  3. Variance Decomposition: Suppose you want to estimate the variance of the daily CTRs across the week. How does the calculation of sample variance ($s^2$) differ from population variance ($_sigma^2$), and why is the distinction important when estimating the stability of your CTR metric?

Example

Assume on Day 1: $n = 1000$, $k = 50$.

  • The Wald 95% confidence interval uses $Z = 1.96$.
  • Standard error $SE = sqrt((p * (1-p)) / n)$.
  • Calculation: $0.05 \pm 1.96 * sqrt((0.05 * 0.95) / 1000) \approx 0.05 \pm 0.0135$.

Hints

Hint 1

Recall that the variance of a binomial proportion is defined as $p(1-p)/n$.

Hint 2

Consider the central limit theorem requirements for the Wald interval; specifically, what happens when $p$ is very close to 0 or 1?

Hint 3

For the variance of the daily averages, remember that the sample variance uses Bessel's correction $(n-1)$ to provide an unbiased estimate of the population variance.

Follow-up questions

  • How would your approach to calculating a confidence interval change if you observed a CTR of 0% on several days?
  • If we observe high day-to-day volatility in CTR, how can we determine if this is due to sampling noise or external factors like platform traffic changes?
  • How does the Wilson score interval improve upon the limitations of the Wald interval?