Katkat
Back

Assess Routing Experiment Validity

WaymoData Scientist · Online Assessment
MediumData ScientistOnline AssessmentStatistics & Math

Last updated

This question evaluates experimental analysis and statistical hypothesis testing skills, including data preparation and joining, interpretation of p-values, and understanding of causal inference and threats to validity in A/B tests.

A ride-hailing team runs an A/B test in San Francisco in July 2024 for a new routing algorithm intended to reduce time to pickup, abbreviated TTP.

You are given two pandas DataFrames:

df_users:

  • user_id: unique user identifier
  • variant: experiment assignment, either control or treatment

df_rides:

  • ride_id: unique ride identifier
  • user_id: user identifier
  • ride_date: ride timestamp or date
  • city: ride city
  • time_to_pickup: numeric TTP in minutes

Tasks:

  1. Write Python code to filter to San Francisco rides in July 2024, join ride records to experiment assignments, run a Welch two-sample t-test comparing treatment versus control TTP, and return the p-value.
  2. Given the returned p-value, how would you decide whether the result is statistically significant?
  3. Is a statistically significant p-value conclusive proof that the new routing algorithm is better? If not, explain the main threats to validity.
  4. Propose a stronger experiment design and analysis plan for this routing algorithm.