Design and Analyze Airbnb Locker Experiment

Airbnb is considering launching a luggage locker feature that lets guests store their bags before their scheduled check-in time, so they don't have to wait until the room is ready. You are the data scientist asked to design — and then analyze — an experiment that decides whether Airbnb should launch this feature.

The interview has two halves: a case-study design discussion (Parts 1–5, 7) and a hands-on analysis of a provided experiment dataset (Part 6).

Constraints & Assumptions

• Airbnb is a two-sided marketplace: an intervention can affect guests, hosts, and the platform's revenue simultaneously, and these interests can conflict.
• Lockers are (at least partly) shared physical infrastructure with finite capacity in a given location.
• Outcomes span the full booking lifecycle: assignment may happen at booking time, but key outcomes (post-stay rating, support contacts, damage claims) only mature after checkout.
• Assume standard significance and power targets unless you argue otherwise: $\alpha = 0.05$ (two-sided) and $80%$ power.
• You have access to historical baselines (e.g., support-contact rate, booking conversion) to power the test.

Clarifying Questions to Ask

• Is the locker offered before booking (a search/booking feature) or after booking (a check-in convenience)? This determines the primary metric and randomization point.
• Are lockers on-listing (host-operated, e.g., a smart lock box) or shared off-site infrastructure with pooled capacity? This determines whether SUTVA can hold at the reservation level.
• What is the intended launch mechanism — a UX toggle, a host opt-in, or a market-wide rollout? The experiment should mirror how it will actually ship.
• Is the feature free or priced? Pricing changes both the metric set and the revenue tradeoff.
• What baseline rates do we have for the candidate primary metric, and what effect size would be worth launching?
• Are there safety, liability, or insurance constraints (lost/damaged luggage) that gate launch regardless of the metric outcome?

Part 1 — Randomization unit

Should the experiment be randomized on the guest side, reservation side, host/listing side, or market side? Justify your choice, and explain how the answer depends on what exactly the treatment changes.

What This Part Should Cover

• Reasons from SUTVA, not from a default ("just randomize users").
• Identifies both contamination channels — shared locker capacity and host-level behavior change — and which unit internalizes each.
• Connects the unit choice to the launch mechanism and acknowledges the power vs. validity tradeoff (finer = more powerful but riskier).

Part 2 — Eligibility, treatment, control

Define the eligible population, the treatment experience, and the control experience precisely enough that an engineer could implement the assignment logic.

What This Part Should Cover

• Eligibility defined pre-assignment and independent of any post-treatment behavior.
• Treatment defined as the offer of locker access, not actual usage.
• Control defined as the realistic status quo (current host-managed workaround), not a strawman.

Part 3 — Metrics

Choose a primary metric, a set of secondary metrics, and guardrail metrics. Discuss the tradeoffs among guest experience, host experience, operational cost, safety, and marketplace revenue.

What This Part Should Cover

• A single, well-justified primary tied to the feature's funnel position.
• A coherent metric tree (primary / secondary / guardrail), not a flat list of everything measurable.
• Explicit marketplace tradeoffs — names the guardrails that protect hosts, safety, and cost, and states a launch decision rule.

Part 4 — Sample size and duration

Explain how you would compute the required sample size and decide the experiment duration.

What This Part Should Cover

• Sample size driven by a pre-registered MDE and the correct two-proportion power formula (for a binary primary).
• Design-effect inflation when randomizing at a cluster, with the right intuition for $\rho$ and cluster size.
• Duration reasoning that goes beyond $n/\text{traffic}$: weekly seasonality, the booking-to-checkout maturation lag, and thin-market constraints.

Part 5 — Analysis plan

Describe how you would estimate the treatment effect and compute a p-value for the primary metric, including how you'd reduce variance and handle clustered randomization.

What This Part Should Cover

• States the ITT estimand explicitly and never conditions on post-treatment usage.
• Matches the test to the metric type and reports a CI / effect size, not just a p-value.
• Variance reduction (covariate adjustment / CUPED) and cluster-robust inference matched to the actual randomization level.

Part 6 — Hands-on A/B and A/A test

You are given a reservation-level experiment table with these columns:

Explain, concretely, how you would (a) run the A/B test to estimate the effect and p-value on this table, and (b) run an A/A test to detect bias, sample-ratio mismatch, or instrumentation issues. Reference the specific columns you'd use.

What This Part Should Cover

• A column-specific, runnable A/B procedure: filter on is_aa_test, validate unit_id uniqueness and that assigned_at precedes outcomes, estimate ITT on the chosen primary.
• Treats used_locker strictly as post-treatment (a diagnostic, never a filter or conditioning variable for the headline).
• An A/A that runs the SRM (chi-square) check and covariate balance via standardized mean differences, and explains what a failure implies for trusting the live A/B.

Part 7 — Risks

Discuss the risks that threaten this specific experiment: selection bias, interference between treatment and control, locker capacity constraints, host-side spillovers, and heterogeneous effects across markets. For each, name how your design or analysis mitigates it.

What This Part Should Cover

• A risk-to-mitigation mapping (each risk paired with the specific design or analysis lever), not a generic risk list.
• Recognizes that the mitigations are the same levers chosen earlier (unit choice, ITT, stratification), showing the design hangs together.
• Calls out heterogeneity discipline — pre-specified stratification, not post-hoc subgroup fishing.