Katkat
Back

Design an Online Experiment

WaymoSoftware Engineer · Onsite
MediumSoftware EngineerOnsiteMachine Learning

Last updated

This question tests the ability to design a statistically sound A/B test for a machine learning model update in a real-world, dynamic environment, focusing on metric selection and mitigation of interference.

Scenario

Waymo is testing a new path-planning model update intended to make autonomous vehicle maneuvers smoother at intersections. You are tasked with designing an online experiment (A/B test) to evaluate if this update reduces 'harsh braking' events without increasing trip duration.

Requirements

  1. Randomization Unit: Propose a strategy for assigning traffic scenarios or vehicles to treatment versus control. Justify your choice regarding the balance between bias and variance.
  2. Metrics: Define one primary metric for safety (the 'harsh braking' goal) and one guardrail metric for efficiency (the 'trip duration' goal).
  3. Interference: Identify a potential source of 'network interference' or 'spillover effect' in this experiment. How would you detect or mitigate this effect?
  4. Seasonality: If the experiment runs for three weeks, how do you handle potential day-of-week effects or specific weather conditions that might skew the results?

Considerations

  • A/B testing in autonomous driving is difficult because agents (the vehicles) interact with a shared environment.
  • Ensure your design accounts for the fact that a 'treatment' vehicle's behavior might influence the behavior of other human-driven vehicles nearby.

Hints

Hint 1

Consider whether assigning individual trips, specific geographical regions, or individual vehicles is the most robust way to ensure independent samples.

Hint 2

Think about the difference between 'user-level' randomization and 'cluster-level' randomization when dealing with traffic flow interactions.

Hint 3

Discuss using 'Difference-in-Differences' (DiD) or pre-experiment normalization to account for systematic variations in traffic patterns across different times of day.

Follow-up questions

  • How would you determine the minimum sample size required for your experiment to achieve a desired power and significance level?
  • If the new model performs well in sunshine but poorly in heavy rain, how would you report your results to stakeholders?
  • What steps would you take if you discovered a significant 'novelty effect' where the vehicle's driving style changed only because it was being monitored by a safety driver?