Katkat
Back

Explain Bootstrap and Statistical Inference

GoogleData Scientist · Technical Screen
HardData ScientistTechnical ScreenStatistics & Math

Last updated

This problem tests your understanding of bootstrap resampling theory, confidence interval construction, and the statistical trade-offs involved in hypothesis testing for small-sample experiments.

Part 1: Bootstrap Mechanics

You are analyzing a metric from a new feature launch with a sample size of n = 50. The data distribution is unknown and slightly skewed. Explain how you would use the bootstrap method to construct a 95% confidence interval for the population mean. Specifically, define the resampling process, explain how the distribution of the bootstrap replicates relates to the sampling distribution of the mean, and note any limitations of this approach when n is very small.

Part 2: Hypothesis Testing and Power

Suppose you are comparing two variants (A and B) using a t-test. You observe a p-value of 0.04.

  1. Interpret what this p-value means in the context of the null hypothesis.
  2. If you decide to repeat this test 20 times on independent sub-samples of your data to 'increase confidence,' what statistical issue arises? How should you adjust your significance level (alpha)?
  3. If your experiment has low statistical power, what is the implication for your findings regarding the Type II error rate (false negatives)?

Part 3: Probability Theory

Let X_1, X_2, ..., X_n be independent and identically distributed random variables drawn from a Uniform distribution U(0, theta), where theta is an unknown parameter. Derive the Maximum Likelihood Estimator (MLE) for theta. Is this estimator biased? If so, how would you construct an unbiased estimator?

Hints

Hint 1

For bootstrapping, remember that you are sampling with replacement from the original sample to create a bootstrap distribution that approximates the sampling distribution of the statistic.

Hint 2

When performing multiple comparisons, consider the family-wise error rate and the impact of the Bonferroni correction on your alpha threshold.

Hint 3

For the MLE of the uniform distribution, realize that the likelihood function increases with theta until the largest observation, at which point the probability drops to zero for any value larger than the max.

Follow-up questions

  • How does the choice of bootstrap confidence interval method (e.g., Percentile vs. BCa) impact accuracy for skewed distributions?
  • If your experiment was part of a larger A/B test suite with 100 concurrent tests, what specific procedures would you implement to control the False Discovery Rate (FDR) instead of just the family-wise error rate?
  • How would the variance of your MLE change if you increased the sample size n, and what is the relationship between this variance and the Fisher Information?