Test Whether a Routing Experiment Reduced Pickup Time

An A/B test in San Francisco during July 2024 evaluates a new routing algorithm intended to reduce time to pickup. You have:

df_users(user_id, variant)
df_rides(ride_id, user_id, ride_date, city, time_to_pickup)

Constraints & Assumptions

• Use Python with pandas and a standard statistical library.
• Filter to San Francisco rides from July 1 through July 31, 2024.
• The requested calculation is a two-sided Welch's t-test comparing pickup time between variants.
• State the independence limitation if users can contribute multiple rides.

Clarifying Questions to Ask

• Was treatment assigned at the user level, and is assignment unique and stable?
• Which variant is control, and are there missing or duplicate assignments?
• Is ride_date timezone aware, and are invalid or canceled rides already excluded?
• What significance level and minimum practically important effect were chosen before analysis?

Part 1: Compute the P-Value

Filter the rides, join users to their assigned variants, run Welch's t-test on time_to_pickup, and return the p-value. Explain the important data-quality checks.

What This Part Should Cover

• Correct half-open date filtering, validated join cardinality, null handling, and equal_var=False.
• Group counts and means alongside the p-value.
• Recognition that ride-level observations may be clustered within users.

Part 2: Interpret and Improve the Experiment

Explain when the result is statistically significant and whether significance alone proves the new algorithm is better. Propose a stronger experiment and analysis.

What This Part Should Cover

• Comparison with a preselected alpha, an effect estimate, confidence interval, and practical significance.
• Checks for randomization integrity, sample-ratio mismatch, interference, attrition, and metric quality.
• An analysis aligned with user-level assignment or clustered repeated rides.
• Primary metrics, guardrails, duration, segmentation, and a launch rule defined in advance.