Welch's t-Test p-Value for a Routing Experiment on Time to Pickup, and Its Validity

This two-part task comes from a timed online assessment for a data science role at a ride service. A new routing algorithm intended to reduce time to pickup (TTP, the time until a vehicle reaches the rider) was tested in San Francisco in July 2024. Users were assigned to variants.

You have two pandas DataFrames:

• df_users with columns user_id, variant;
• df_rides with columns ride_id, ride_date, city, time_to_pickup.

Assume df_rides also carries the rider's user_id, which the join to df_users requires, and that San Francisco appears in city under a single label such as "SF"; confirm both before relying on them.

Clarifying Questions

• What are the variant labels, and is every user assigned to exactly one variant?
• Should the test be two-sided, or one-sided in the direction of lower TTP?
• What significance level should be used?
• How should rides with a missing time_to_pickup be treated?
• Is ride_date a date or a timestamp, and in which time zone is "July" defined?

Part 1 — Compute the p-value

Write a Python function that filters the rides to San Francisco in July 2024, joins each ride to its rider's variant, runs Welch's t-test comparing time_to_pickup between the two variants, and returns the p-value.

What This Part Should Cover

• Correct filtering by city and by the full month, with date parsing and an exclusive end boundary
• A join that neither duplicates nor silently drops rides, with a check for users in more than one variant
• Welch's t-test (unequal variances) with the right inputs, and handling of missing values
• What the test statistic and its degrees of freedom are

Part 2 — Experiment validity

Now interpret the result. Is it statistically significant at your chosen level? Can the team conclude that the new routing algorithm reduces TTP? If not, design a better experiment.

What This Part Should Cover

• Reading the p-value correctly, together with the direction and size of the effect
• Threats to validity: mismatch between the randomization unit and the analysis unit, interference between variants through the shared fleet, and limits of one city and one month
• A better design (for example switchback or geographic randomization), with its metrics, duration, power and analysis plan