Katkat
Back

Evaluate a cold-start rating launch

UberData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests the ability to design an evaluation framework for a new feature in a two-sided marketplace while accounting for spillover effects and metrics that represent different stakeholder interests.

Scenario

Uber is launching a "Priority Matching" feature in a specific city. The feature allows riders to pay an extra fee to be prioritized in the matching queue, effectively reducing their wait times. As the Data Scientist lead, you are tasked with evaluating the impact of this launch.

The Challenge

You have decided to run an A/B test by randomly assigning users into two groups: Treatment (can use Priority Matching) and Control (cannot). However, your stakeholders are concerned that the feature might negatively impact the control group by increasing their wait times (as priority riders "cut" the line).

  1. Propose a metric framework to evaluate the success of this launch. What are your primary, guardrail, and counter-metrics?
  2. Discuss the potential for network interference (spillover effects) in this experiment. How does the presence of priority riders change the experience for non-priority riders and drivers?
  3. How would you design the experiment or the analysis to account for these spillover effects if standard randomization at the user level is insufficient?
  4. Suppose the data shows an increase in net revenue but a decrease in overall platform satisfaction scores for the control group. How would you present this trade-off to product leadership?

Hints

Hint 1

Consider metrics that cover both sides of the marketplace, such as rider wait times, driver earnings per hour, and overall booking completion rates.

Hint 2

Think about the difference between randomization at the individual user level versus geo-spatial or time-based (switchback) randomization.

Hint 3

When evaluating trade-offs, focus on long-term ecosystem health versus short-term revenue gains and how to quantify "fairness" in queue management.

Follow-up questions

  • How would you adjust your measurement strategy if the priority matching feature creates high volatility in driver earnings across different neighborhoods?
  • If you decide to use a switchback (time-based) experiment instead of A/B testing, what are the primary risks and how do you mitigate carryover effects?
  • How would you determine if the revenue gains from Priority Matching are truly incremental or simply cannibalizing existing high-frequency rider spend?