Katkat
Back

Diagnose post-release conversion regression rigorously

AppleData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests the ability to diagnose a post-release metric regression in the absence of a control group by utilizing quasi-experimental design and causal inference methods.

Scenario

Your team released version 7.3 of the mobile app last Tuesday. Since then, the global signup conversion rate (Signups / Total Sessions) has dropped from 5.4% to 4.9%. Stakeholders are panicking, but the engineering team claims no breaking code changes were introduced.

Because the update was a forced 100% rollout, you have no concurrent A/B test control group. You suspect the drop might be a genuine bug, a seasonal fluctuation, a marketing campaign shift, or a change in bot traffic patterns.

Your Task

  1. Propose a methodological framework to identify the root cause of this 50-basis-point drop.
  2. Explain how you would rule out external factors like marketing, seasonality, or technical tracking errors.
  3. Since you lack a direct control group, describe how you would construct a synthetic control or counterfactual to quantify the impact of the version upgrade.
  4. Provide a clear decision framework: Under what conditions would you recommend a rollback versus a hotfix or further investigation?

Hints

Hint 1

Consider looking at conversion trends segmented by platform (iOS vs. Android) or app version if some users haven't updated yet.

Hint 2

Think about using Difference-in-Differences (DiD) or Causal Impact analysis by comparing the post-release period to a similar historical window or a control group of users that did not receive the update (if any).

Hint 3

Analyze the conversion funnel steps individually to determine if the drop is localized to a specific UI interaction or occurring globally across all entry points.

Follow-up questions

  • How would you account for 'day-of-week' effects if the release occurred on a Tuesday and you are comparing it to the previous week?
  • If you find that bot traffic spiked simultaneously with the release, how would you filter this to ensure the analysis reflects human behavior?
  • What metrics would you prioritize tracking in the future to ensure you can detect such regressions before a 100% rollout?