Katkat
Back

Compare Candy-Color Guessing Experiments with Different Sample Sizes

HeadlandsData Scientist · Technical Screen
MediumData ScientistTechnical ScreenStatistics & Math

Last updated

Evaluate and compare evidence from two binomial experiments with different sample sizes and hit rates against a random guessing null hypothesis.

Scenario

A subject claims to have an extrasensory or taste-based ability to identify the color of a candy without seeing it. There are 5 distinct colors, uniformly distributed and chosen independently with replacement. Under the null hypothesis of random guessing, the probability of guessing the correct color on any single trial is p0 = 1/5 = 0.20.

You run two separate experiments:

  • Experiment A: The subject attempts n = 3 trials and gets k = 2 correct.
  • Experiment B: The subject attempts n = 100 trials and gets k = 40 correct.

Tasks

  1. Formulate the Hypothesis Test

    • State the null and alternative hypotheses for a one-sided test for both experiments.
    • Explicitly list the key assumptions required for a binomial model to hold in this setup.

  2. Calculate p-values

    • Compute the exact one-sided p-value for Experiment A: P(X >= 2 | n = 3, p = 0.20).
    • Compute or approximate the one-sided p-value for Experiment B: P(Y >= 40 | n = 100, p = 0.20). If using a normal approximation, indicate whether you apply a continuity correction and show the resulting z-score.

  3. Comparative Analysis & Interpretation

    • Experiment A achieved an apparent success rate of 66.7%, whereas Experiment B achieved only 40.0%. Compare the statistical evidence provided by each experiment at a significance level of alpha = 0.05.
    • Explain how sample size affects the tension between effect size and statistical significance in this decision context.

Hints

Hint 1

Recall that the p-value is the probability of observing a result at least as extreme as the actual observation under the null distribution.

Hint 2

For Experiment A, compute the exact probabilities for k = 2 and k = 3 using the binomial probability mass function.

Hint 3

For Experiment B, check whether np and n(1 - p) are large enough to justify a normal approximation, and consider how a continuity correction adjusts the threshold.

Follow-up questions

  • How would your conclusion change if the 5 candy colors were drawn without replacement from a finite bowl of 20 candies?
  • If you adopted a Bayesian approach with a Beta prior on p, how would the posterior distribution for p behave across the two experiments?
  • What sample size would Experiment A require to reject the null at alpha = 0.05 if the observed hit rate remained strictly at 66.7%?