Katkat
Back

Recommend a Decision After a Nonsignificant Experiment

YelpData Scientist · Technical Screen
EasyData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests the ability to interpret nonsignificant experimental results and formulate data-driven business recommendations beyond simple hypothesis testing.

Scenario

You are a Data Scientist at Yelp. Your team launched an A/B test for a new 'Personalized Restaurant Recommendation' widget on the homepage. The goal was to increase the Click-Through Rate (CTR) of listed restaurants.

After running the experiment for two weeks, you observe the following results:

  • Control Group CTR: 4.2%
  • Treatment Group CTR: 4.3%
  • P-value: 0.18
  • 95% Confidence Interval for the lift: [-0.15%, +0.35%]

Task

  1. Explain what these results imply regarding the statistical significance of the experiment.
  2. Given that the experiment did not reach statistical significance at the alpha = 0.05 level, what is your recommendation: launch, roll back, or continue testing? Justify your decision.
  3. Identify at least two non-statistical factors you would consider before making a final product decision.

Constraints

  • Assume the sample size was sufficiently powered to detect a 0.2% absolute lift at 80% power.
  • Consider the practical implications of the observed interval rather than just the p-value.
  • Your response should focus on data-driven decision-making processes rather than just the raw math.

Hints

Hint 1

Consider whether a p-value of 0.18 suggests there is no effect at all or simply that the effect is too small to distinguish from noise given the current sample size.

Hint 2

Evaluate the confidence interval: does the range include values that would be considered 'practically significant' for business metrics?

Hint 3

Think about the cost of maintaining the new feature versus the potential upside of a small but positive incremental lift.

Follow-up questions

  • If the business insists on launching to gain even a marginal advantage, how would you mitigate risks after the launch?
  • How might you redesign the experiment or segment the data if you suspect the null result is hiding a positive impact on a specific sub-population?
  • What metrics other than CTR should we monitor if we decide to roll out this feature despite the nonsignificant primary result?