Katkat
Back

Design and assess an A/B test

AirbnbData Scientist · Onsite
HardData ScientistOnsiteAnalytics & Experimentation

Last updated

This problem tests your ability to design a statistically sound A/B test for a marketplace platform, including metric selection, power analysis considerations, and handling potential interference effects.

Scenario

Airbnb is considering a new "Instant Book" badge that appears on listings for hosts who meet specific high-responsiveness criteria. The product team believes this will increase the conversion rate from "search" to "booking."

Your Task

Design an experiment to evaluate the impact of this badge. Provide a structured response covering the following:

  1. Metric Definition: Define your Primary metric, Secondary metric(s), and Guardrail metric(s).
  2. Unit of Randomization: Choose the unit of randomization (e.g., user, listing, or geographic region). Justify your choice, specifically addressing potential network effects or interference.
  3. Power Analysis: If the baseline conversion rate is 5% and you want to detect a 2% relative lift with 80% power and a 5% significance level, explain how you would estimate the required sample size.
  4. Segmentation: Identify key segments you would analyze to ensure the feature doesn't disproportionately hurt specific user groups.

Constraints

  • Assume the platform has millions of monthly active users.
  • Consider that "Instant Book" listings might cannibalize "Request to Book" listings.
  • Your answer should demonstrate an understanding of the trade-offs between speed, precision, and business risk.

Hints

Hint 1

Consider the risk of 'spillover effects' where listings in the control group are affected by the treatment group's behavior.

Hint 2

Think about how 'Guardrail' metrics are essential to protect the long-term health of the marketplace, such as host cancellation rates or support ticket volume.

Hint 3

For sample size estimation, remember that the variance of a proportion depends on the mean, which will impact your calculation for conversion rate.

Follow-up questions

  • How would you handle the situation where the experiment shows a statistically significant increase in bookings but a significant decrease in host satisfaction?
  • If we wanted to test this feature only in specific markets (e.g., London vs. Paris), how would you structure the experiment to avoid bias?
  • What statistical methods would you use to account for the fact that listing-level randomization might lead to 'interference' where a treated listing takes demand from a control listing?