Katkat
Back

How would you estimate impact without A/B?

MicrosoftData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests your ability to design quasi-experimental methods to infer causal impact when randomization is not possible due to technical or business constraints.

The Scenario

You are a Data Scientist at a cloud productivity company. The product team has just launched a new 'Smart Summary' feature for all enterprise users globally to help them catch up on missed meetings.

Because the feature was deployed as a mandatory update to ensure security compliance, there was no control group, and no A/B test was conducted. The Product Manager observes a 15% increase in 'User Engagement' (measured by time spent in the app) since the launch but is worried this might just be a seasonal trend or user novelty effect.

Your Task

Describe how you would rigorously estimate the causal impact of the 'Smart Summary' feature on 'User Engagement'. Specifically:

  1. Propose two quasi-experimental methods you would use to control for external factors or selection bias.
  2. Explain how you would validate the 'parallel trends' assumption if you chose a Difference-in-Differences (DiD) approach.
  3. Discuss how you would address potential 'novelty effects' or 'learning curves' that might inflate the observed impact in the first few weeks.

Provide your answer assuming you have access to historical user activity logs, company-wide seasonal trends, and usage data from similar feature releases in the past.

Hints

Hint 1

Consider using a synthetic control method or a control group from a region where the feature rollout was delayed.

Hint 2

Think about how you could use propensity score matching to construct a pseudo-control group from historical data.

Hint 3

Address the temporal aspect by analyzing the decay of the engagement lift over time to distinguish between long-term utility and initial curiosity.

Follow-up questions

  • How would you detect or mitigate potential interference, such as network effects where users who don't have the feature are impacted by users who do?
  • What steps would you take to ensure that the observed increase isn't just Simpson's Paradox occurring across different segments of power users?
  • If you had to design a 'Holdout' experiment retroactively, how would you justify the potential churn risk to stakeholders?