Evaluate a New Version in Simulation and Explain a Flat Live Result
WaymoData Scientist · Technical Screen
Last updated
Compare model versions on paired simulation scenarios and diagnose why offline improvements may disappear in live rare-event evaluation.
Evaluate a New Version in Simulation and Explain a Flat Live Result
A new version performs better in offline simulation but shows no detectable improvement after launch. You also have historical scenarios that can be replayed under both the old and new versions. Explain how you would compare the versions using these scenarios and investigate the disagreement with live results, especially when the outcomes of interest are rare events.
Hints
Hint 1
Use the same scenario twice: A scenario that is intrinsically difficult can produce correlated outcomes under both versions.
Follow-up questions
- What if the simulation scenarios deliberately oversample hazardous situations?
- Why is a nonsignificant live comparison not evidence that the versions are equivalent?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.