Test whether two distributions differ

You have two datasets collected from different systems or populations, and you want to determine whether they come from the same distribution.

Explain how you would test distribution equality in univariate and multivariate settings. Also describe common sampling methods useful for collecting evaluation data.

Constraints & Assumptions

• Define the null and alternative hypotheses.
• Choose tests based on variable type and whether the question is about means, proportions, tails, or full distributions.
• Discuss assumptions such as independence, clustering, and sample comparability.
• Pair statistical significance with practical significance.
• Include multiple-testing, power, and sampling-design considerations.

Clarifying Questions to Ask

• What decision will this test support?
• Are observations independent or clustered by user, vehicle, route, time, or city?
• Are the two samples supposed to represent the same target population?
• Are we comparing one feature or many?
• Do tails or rare events matter more than averages?

Part 1 - Define Hypotheses And Run Univariate Tests

How would you test distribution equality for one variable?

What This Part Should Cover

• H0: P = Q versus H1: P != Q.
• Continuous tests such as KS, Anderson-Darling, Wasserstein or energy distance with permutation.
• Categorical tests such as chi-squared or Fisher's exact test.
• Mean-only tests such as t-test or Welch's t-test when the question is narrower.

Part 2 - Handle Multivariate Comparisons

How would you compare high-dimensional samples?

What This Part Should Cover

• Classifier two-sample tests, MMD, energy distance, permutation tests, Hotelling's T-squared for mean vectors under stronger assumptions, and feature-level diagnostics.

Part 3 - Interpret Results And Practical Issues

How would you handle statistical significance, practical significance, sample size, power, multiple testing, and dependence?

What This Part Should Cover

• Effect sizes, confidence intervals, business thresholds, tail metrics, Bonferroni or Benjamini-Hochberg, clustered bootstrap, and target-population comparability.

Part 4 - Sampling Methods

What sampling methods are useful for evaluation data?

What This Part Should Cover

• Simple random, stratified, cluster, multistage, systematic, importance or weighted, reservoir, and convenience sampling.
• When to prefer stratified or oversample-then-reweight designs.