Katkat
Back

Design experiments for payments, search, and promotions

DoorDashData Scientist · Onsite
EasyData ScientistOnsiteAnalytics & Experimentation

Last updated

This question tests your ability to design robust A/B tests for marketplace products, focusing on metric selection, identifying interference, and accounting for ecosystem-level trade-offs.

Scenario

You are a Data Scientist at a food delivery platform. You are tasked with designing experiments for three distinct features. For each, describe your experimental design, your primary metric, and how you would address potential cross-side interference.

1. Payment Methods

The Product team wants to introduce a 'Buy Now, Pay Later' (BNPL) option at checkout. They expect this to increase conversion rates.

2. Search Relevance

The Search team is testing a new ranking algorithm that prioritizes restaurants with faster delivery speeds over those with higher historical ratings.

3. Promotions

The Marketing team wants to offer a 'Spend $25, Get $5 Off' coupon to a subset of users to increase order frequency.

Requirements

For each case, address:

  • Metric Framework: What is your Primary metric and one Guardrail metric?
  • Experimental Unit: What is your randomization unit (e.g., user, restaurant, city)?
  • Interference: How will you handle network effects or market-level spillovers?

Hints

Hint 1

Consider whether the experimental unit should be the user or the geographic region to minimize spillover effects.

Hint 2

Guardrail metrics are essential for ensuring that short-term gains in conversion do not cause long-term harm to restaurant churn or delivery partner availability.

Hint 3

Think about how 'Buy Now, Pay Later' might change user behavior in a way that affects other payment processors or overall order volume.

Follow-up questions

  • How would you measure the cannibalization effect of the promotion on organic orders?
  • If the search ranking change leads to a 5% increase in orders but a 2% increase in delivery driver wait times, how would you determine if the overall impact is positive?
  • What statistical methods would you use to account for violations of the Stable Unit Treatment Value Assumption (SUTVA) in a two-sided marketplace?