Katkat
Back

How validate a driving simulation is realistic?

WaymoData Scientist · Onsite
EasyData ScientistOnsiteAnalytics & Experimentation

Last updated

This question tests the ability to design an evaluation framework to compare simulated vehicle behavior against real-world sensor and trajectory logs to ensure high-fidelity simulation.

The Problem

Waymo uses driving simulations to test autonomous vehicle software before deployment. As a Data Scientist, you are tasked with validating that a newly developed 'Rainy Night' simulation environment is realistic. You have access to a dataset containing 1,000 real-world driving logs (ground truth) and 1,000 simulated logs where the autonomous vehicle navigated the same geographic area under identical environmental conditions.

Your goal is to propose a framework to determine if the simulation is 'realistic' enough to replace road testing for these scenarios. Address the following:

  1. Define three key metrics that capture the behavior of the autonomous vehicle (e.g., trajectory, braking patterns, or reaction times).
  2. Describe the statistical methodology you would use to compare the real-world metric distributions against the simulated ones.
  3. How would you handle the fact that simulations might not perfectly mirror the noise or sensor artifacts found in real-world rainy night data?

Example

If you choose 'Average Lateral Deviation' as a metric, you might compare the mean and variance of deviations from the lane center. If the simulation shows a significantly lower variance than real data, it suggests the simulation is 'too perfect' and lacks the edge-case variations (like hydroplaning or sensor occlusion) present in reality.

Hints

Hint 1

Consider using Kolmogorov-Smirnov tests or Wasserstein distance to compare the probability distributions of your metrics.

Hint 2

Think about 'scenario slicing'—does the simulation match real-world behavior equally well at low speeds versus high speeds?

Hint 3

Consider whether the goal is for the simulation to be 'identical' to reality or if it should be 'stochastically representative' of the underlying risks.

Follow-up questions

  • If the simulation is statistically different from real-world data, how do you determine which specific behaviors (e.g., turn signal timing, acceleration curve) are causing the divergence?
  • How would you account for 'simulation bias,' where the vehicle software behaves differently because it knows it is in a simulation versus on the road?
  • What would you do if the simulation provides 100% safety, but real-world data shows a 0.5% collision rate in similar conditions?