Katkat
Back

Choose tests under non‑normal, unequal variance

InstacartData Scientist · Onsite
HardData ScientistOnsiteStatistics & Math

Last updated

This question tests the ability to select and justify statistical inference methods for metrics that violate standard parametric assumptions, specifically when dealing with heavy-tailed, heteroskedastic, and zero-inflated distributions.

Scenario

You are evaluating an A/B test for a new feature on the Instacart platform designed to increase the 'Total Order Value' per user session. After running the experiment, you observe the following characteristics in the primary metric:

  1. Zero-Inflation: A significant portion (approx. 30%) of users have a value of $0$ because they logged in but did not purchase.
  2. Heavy-tailedness: The distribution of non-zero order values is strongly right-skewed with a few extreme 'power users'.
  3. Heteroskedasticity: The treatment group shows significantly higher variance in order values compared to the control group.

Task

Explain why a standard independent two-sample t-test might yield unreliable p-values and confidence intervals in this scenario. Then, propose and justify two alternative approaches to perform inference. For each alternative, detail how you would handle the zero-inflation and the skewness, and discuss how you would interpret the resulting effect size compared to the raw mean difference.

Hints

Hint 1

Consider how the violation of the normality assumption affects the Type I error rate when the sample size is moderate and the underlying distribution is highly skewed.

Hint 2

Evaluate the utility of a two-part (hurdle) model versus a log-transformation, and consider the implications of back-transforming log-normal estimates to the original scale.

Hint 3

Think about the role of the Central Limit Theorem in the context of the variance-mean relationship and whether bootstrapping or permutation tests could provide a more robust estimation of the sampling distribution.

Follow-up questions

  • How would your approach change if the business stakeholder specifically requested a measure of the 'median impact' instead of the mean impact?
  • If you decide to use a log-transformation, how do you handle the zero values before applying the transform?
  • How does the choice of test impact the power of the experiment, specifically given the presence of extreme outliers?