Katkat
Back

Design an Unbiased Upgrade Experiment

GoogleData Scientist · Technical Screen
HardData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests the ability to design an A/B test for a software upgrade, specifically focusing on handling non-compliance, selection bias, and the identification of causal effects in a real-world product environment.

Scenario

You are a Data Scientist at a cloud storage company. Your team has developed a significant performance upgrade for the desktop sync client. Because the upgrade requires a system reboot, users must manually click 'Install Update' to move from v1.0 to v2.0.

Product management wants to know the impact of the v2.0 upgrade on 'Retention Rate' (defined as a user remaining active for 30 days post-intervention). You observe that users who choose to upgrade voluntarily are typically your most power-using, loyal customers, while those who do not upgrade are often less engaged.

The Task

  1. Explain why a simple comparison of Retention Rate between users who upgraded and those who stayed on v1.0 is biased.
  2. Design an experimental framework that allows you to estimate the causal impact of the upgrade on Retention Rate.
  3. Define the metrics you would monitor to ensure the experiment is valid and the results are interpretable.

Constraints

  • You cannot force the update on all users due to technical stability risks.
  • The 'selection effect' (i.e., loyal users being more likely to click the button) is significant.
  • Assume you have a massive user base (millions of daily active users) and the ability to randomly bucket users into groups.

Hints

Hint 1

Consider how a randomized controlled trial (RCT) using an 'encouragement design' could isolate the causal effect without forcing an upgrade on everyone.

Hint 2

Think about the difference between the Intent-to-Treat (ITT) effect and the Treatment-on-the-Treated (TOT) effect in the presence of non-compliance.

Hint 3

Look into the Wald estimator as a method to recover the average treatment effect for the subgroup of users who would comply with the update.

Follow-up questions

  • How would you account for 'novelty effects' or 'learning curves' that might influence retention immediately after the update?
  • If the upgrade caused a temporary spike in server load, how might that interfere with your ability to measure the 'true' impact on retention?
  • What steps would you take to ensure the 'encouragement' mechanism itself does not influence retention, independently of the software upgrade?