Katkat
Back

Assess free-month promotion impact

OpenAIData Scientist · Technical Screen
HardData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests your ability to design a rigorous experimentation framework to measure the causal impact of a promotional offer while accounting for self-selection bias and long-term churn behavior.

Scenario

OpenAI is planning to offer a 'one month free' promotion for ChatGPT Plus to a subset of existing free-tier users to increase long-term conversion and revenue. The Product team wants to run an A/B test by randomly assigning users to a treatment group (offer) and a control group (no offer).

The Challenge

  1. You are tasked with designing the experiment. How would you define the primary and secondary metrics to determine if this promotion is a 'success'?
  2. Given that users are aware of the promotion, they may intentionally delay upgrading to wait for such offers. How do you quantify this 'cannibalization' effect or 'wait-and-see' bias?
  3. If you find that the treatment group has a higher upgrade rate during the month but a similar churn rate to the control group three months later, how do you calculate the actual ROI of the promotion?

Key Considerations

  • Data: You have access to user session logs, subscription events, and historical engagement data (e.g., messages sent per day).
  • Constraints: The marketing budget is limited, and the engineering team can only support a 10% traffic split for the treatment group.
  • Edge Cases: How do you handle 'power users' who were already highly likely to convert without the promotion versus 'dormant' users who might only engage during the free month?

Hints

Hint 1

Consider defining metrics beyond just 'conversion rate,' such as 'Customer Lifetime Value (CLTV)' or 'net incremental revenue' to account for both the discount cost and potential future subscriptions.

Hint 2

Think about using a Difference-in-Differences (DiD) approach or a synthetic control group to isolate the treatment effect from the pre-existing intent of users.

Hint 3

Address the selection bias by segmenting users based on their historical engagement intensity before the experiment begins to see if the promotion is actually attracting new value or just subsidizing existing demand.

Follow-up questions

  • How would you adjust your analysis if you realized the treatment group started sharing their 'free' status with the control group (network effects/spillover)?
  • What statistical tests would you use to ensure the results are robust, especially if the distribution of revenue is highly skewed by a few 'whale' users?
  • If the experiment results are positive, how do you decide on the optimal 'threshold' of engagement for a user to be eligible for future promotional offers?