Katkat
Back

Answer core probability and inference questions

AmazonData Scientist · Technical Screen
EasyData ScientistTechnical ScreenStatistics & Math

Last updated

Tests fundamental statistical inference and probability concepts essential for experimentation and data analysis, including the Central Limit Theorem, hypothesis testing errors, and causal reasoning.

Scenario

As a Data Scientist evaluating an A/B test on Amazon's checkout funnel, you need to clearly articulate statistical principles to both technical teammates and product managers. Answer the following four core questions concisely and precisely:

Part 1: Central Limit Theorem (CLT)

State the Central Limit Theorem in your own words. Why is it useful when analyzing metrics like Average Order Value (AOV), which is typically right-skewed?

Part 2: Hypothesis Testing Errors & Trade-offs

Define Type I error (alpha) and Type II error (beta). How are statistical power and significance level mathematically related to these errors? Explain the practical risk of committing a Type I versus a Type II error when launching a new feature.

Part 3: Interpreting p-values & Sample Size

A junior analyst states: "The p-value is 0.03, which means there is a 97% probability that our new checkout button is genuinely better than the control." Correct their interpretation. Additionally, explain what happens to the p-value when sample size grows arbitrarily large even if the true effect size is negligibly small.

Part 4: Correlation vs. Causation

Data shows customers who use the Wishlist feature spend 40% more annually than customers who do not. Can you conclude that encouraging Wishlist usage causes higher spend? Outline two plausible confounders and state what method you would use to establish causality.

Hints

Hint 1

Recall that the Central Limit Theorem applies to the distribution of sample means, not the underlying individual observations.

Hint 2

A p-value is conditional on the null hypothesis being true: P(Data as or more extreme | H0).

Hint 3

Think about user intent or purchase frequency as confounding variables when evaluating feature usage and spend.

Follow-up questions

  • How would you compute the required sample size before launching this A/B test, and what parameters are needed?
  • If you evaluate 20 different metric variants simultaneously at alpha = 0.05 without correction, what is the probability of at least one false positive?
  • What approaches (e.g., CUPED, stratified sampling) can you use to reduce variance and detect smaller effects faster?