Katkat
Back

Walk Through an Experiment From Design to Decision

AirbnbData Scientist · HR Screen
MediumData ScientistHR ScreenAnalytics & Experimentation

Last updated

Walk through an end-to-end A/B test for an online marketplace feature, covering metric definition, randomization design, SRM diagnosis, and making a launch decision with conflicting trade-offs.

Scenario

Airbnb's Guest Experience team wants to increase booking conversion for flexible travelers. The proposed feature is a "Flexible Date Matrix" displayed on search result pages, showing prices and availability across a +/- 3-day window around the guest's selected dates.

You are the lead Data Scientist tasked with designing the experiment, monitoring its execution, and presenting a launch recommendation to product leadership.

Tasks to Address

1. Experiment Design & Randomization

  • State a testable hypothesis linking the feature to user behavior.
  • Select the unit of randomization (e.g., guest user ID, cookie/device, market/city, or listing). Explain the trade-offs regarding interference, network effects, and user experience consistency across devices.
  • Define how you would determine the sample size and experiment duration, accounting for weekly seasonality.

2. Metrics & Instrumentation

  • Primary Metric: Define 1-2 metrics that evaluate direct success.
  • Guardrail Metrics: Identify at least two operational, host-side, or business health metrics that must not degrade (e.g., host cancellation rate, search latency, customer support contacts).
  • Instrumentation & Integrity: Explain how you would verify data logging and check for Sample Ratio Mismatch (SRM) before analyzing results.

3. Evaluating Results & Trade-Offs

Suppose the 14-day experiment finishes with 2,000,000 total visitors randomized 50/50, and you observe the following results:

  • Observed split: 994,200 Control vs. 1,005,800 Treatment.
  • Search-to-booking conversion rate: +1.6% relative lift (p = 0.008).
  • Average Booking Value (ABV): -2.1% relative drop (p = 0.03).
  • 7-day Host cancellation rate: +0.4 percentage points (p = 0.04).
  • Page load latency (p95): +140ms (p < 0.001).

Walk through your analysis:

  1. Check for SRM. Is the observed traffic split acceptable given a 50/50 target? What steps do you take if SRM is detected?
  2. Assess the statistical and practical significance of the conflicting outcomes (higher booking volume vs. lower basket size and higher cancellations).
  3. Which heterogeneous treatment effects or user segmentations would you investigate to understand why ABV and host cancellations moved in these directions?
  4. What is your final launch recommendation (Launch, Do Not Launch, or Iterate), and what follow-up experiment would you prioritize?

Hints

Hint 1

Use a chi-square goodness-of-fit test on the sample counts to evaluate whether the sample ratio mismatch indicates a logging or trigger bug.

Hint 2

Consider two-sided marketplace dynamics: guests optimizing for cheaper dates can reduce average booking value and lead to friction with host expectations or availability.

Hint 3

Evaluate Net Revenue (conversion lift times ABV change minus cancellation costs) rather than looking at conversion rate in isolation.

Follow-up questions

  • If the SRM check fails, what diagnostic queries would you run first to pinpoint where the imbalance was introduced?
  • How would your randomization unit change if listings in high-demand markets had limited availability, creating strong competition between treatment and control guests?
  • How would you handle non-normal distributions and extreme outliers when calculating confidence intervals for Average Booking Value?