Katkat
Back

Reserving an Elevator for Food Deliveries

CoupangData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question evaluates your ability to design an A/B test for an operational policy change in a high-traffic environment, focusing on cluster randomization, metric selection, and handling network effects.

Scenario

Coupang is considering a new policy: during peak hours, one elevator in high-rise buildings will be 'reserved' for delivery couriers to reduce delivery times. However, residents are concerned that this will increase their wait times. You are tasked with designing an experiment to measure the impact of this policy.

The Challenge

  1. Propose an experimental design. Since you cannot easily change elevator software for single residents within a building, justify your choice of randomization unit (e.g., individual, building, or time-window).
  2. Define the primary success metric for the courier side and the primary guardrail metric for the resident side.
  3. Identify a potential source of bias or spillover effect in your design and explain how you would mitigate it.

Constraints

  • Assume you have access to IoT logs for elevator arrivals/departures and delivery completion timestamps.
  • Consider the 'network effect': if couriers get faster, more orders might be placed, potentially changing the baseline usage patterns.

Example

If you choose a switchback design (alternating policy by time slots), explain how you would handle the 'carryover effect' where a long wait time in one slot might influence user behavior in the next slot.

Hints

Hint 1

Consider whether individual elevators or entire buildings serve as the most appropriate unit for cluster randomization to avoid interference.

Hint 2

Think about 'time-of-day' variations and how a switchback design might require a 'washout period' to mitigate transition noise.

Hint 3

Focus on how to differentiate between the 'treatment effect' of the policy and the underlying 'organic growth' of delivery volume.

Follow-up questions

  • How would you determine the minimum detectable effect (MDE) for your primary metric given the high variance in elevator wait times?
  • If the experiment results show a significant improvement for couriers but a neutral impact for residents, what secondary analyses would you perform to ensure the results are robust across different building types (e.g., high-rise vs. low-rise)?
  • How would you account for seasonal factors, such as extreme weather, which might simultaneously increase delivery volume and elevator demand?