Katkat
Back

Evaluate UberEATS priority delivery and membership

UberData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests the ability to design an experiment for a marketplace feature and evaluate its impact on key business metrics like conversion, frequency, and total contribution margin.

Scenario

UberEats is considering launching a 'Priority Delivery' subscription tier where members pay a flat monthly fee to guarantee faster delivery times by prioritizing their orders in the courier dispatch queue. As a Data Scientist, you are tasked with evaluating the efficacy of this program.

Task

  1. Propose a metric framework to evaluate the success of the Priority Delivery program. Distinguish between guardrail metrics, primary conversion metrics, and long-term ecosystem health metrics.
  2. Describe an experimental design to measure the incremental lift of this program. How would you handle potential cannibalization of existing non-priority orders?
  3. How would you estimate the 'incremental contribution margin' per subscriber, considering that prioritizing some orders may increase the estimated time of arrival (ETA) for non-priority users?

Key Considerations

  • Marketplace externalities (network effects).
  • The trade-off between user satisfaction for subscribers vs. non-subscribers.
  • Methods for calculating the true counterfactual in a high-density, real-time marketplace.

Hints

Hint 1

Consider that marketplace supply is finite; if you prioritize one order, you are inherently deprioritizing another, creating a zero-sum game in the short term.

Hint 2

Think about the difference between measuring user-level lift and marketplace-level cannibalization using geo-based randomization (switchback) instead of user-level randomization.

Hint 3

Focus your margin calculation on the delta in courier utilization costs and the retention value of subscribers versus the churn risk of standard users experiencing longer wait times.

Follow-up questions

  • If the experiment shows a net positive for subscribers but a slight decrease in overall order frequency, how would you decide whether to roll out the feature?
  • How would you account for 'novelty effects' where users sign up for the priority tier initially but decrease usage once the perceived value stabilizes?
  • If the priority delivery service creates a 'two-tier' system, what fairness or operational metrics would you track to ensure we aren't alienating long-term loyal non-subscribers?