Katkat
Back

Compute robust inference under skew and outliers

VoleonData Scientist · Technical Screen
HardData ScientistTechnical ScreenStatistics & Math

Last updated

This problem tests your ability to design a robust statistical framework for comparing two groups with highly skewed, heavy-tailed data and potential outliers, moving beyond simple t-tests.

Problem Statement

You are evaluating the performance of a new bidding algorithm (Treatment) versus the existing one (Control) in an online auction marketplace. The primary metric is 'Revenue per User,' which is notoriously non-normal: it contains a high density of zeros, significant right-skew due to 'whale' users, and intermittent extreme outliers caused by system latency spikes.

You have two samples, control and treatment, each containing n observations. You must propose an inference strategy to determine if the mean difference between these groups is statistically significant.

Your Task:

  1. Describe why a standard two-sample t-test is likely to fail in this environment (consider both Type I error rates and power).
  2. Propose a robust estimator for the difference in means and explain how you would construct a 95% confidence interval for it.
  3. Discuss how you would handle the 'whale' outliers: should they be trimmed, winsorized, or transformed? How does your choice impact the sensitivity of the test?
  4. If you chose a transformation (e.g., log or Box-Cox), explain the bias introduced when back-transforming to the original scale and how to correct for it.

Constraints & Context

  • The sample size n is 10,000 per group.
  • Outliers are not necessarily erroneous; they represent legitimate, albeit rare, high-value auction wins.
  • Assume you must maintain a strict False Discovery Rate (FDR) if this experiment is part of a larger multivariate test suite.

Hints

Hint 1

Consider how the Central Limit Theorem interacts with heavy-tailed distributions and whether 10,000 samples are sufficient to guarantee normality of the sample mean.

Hint 2

Look into M-estimators or influence functions as a way to down-weight extreme observations without discarding the information they contain.

Hint 3

Think about the difference between the 'mean of the logs' and the 'log of the mean,' and how a re-transformation correction factor (like the Duan smearing estimator) might be applied.

Follow-up questions

  • How would your approach change if the data were not only skewed but also exhibited heteroskedasticity (different variances) between groups?
  • Explain how you would compute the power of your proposed test if you cannot rely on analytical derivations.
  • If you utilize a non-parametric method like the permutation test, how do you adjust it to test for a difference in means rather than a difference in distributions?