Katkat
Back

Measure free-month promotion impact

OpenAIData Scientist · Technical Screen
HardData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests the ability to design an experiment for a subscription product, focusing on causal inference, handling right-censoring in cohort data, and calculating the Return on Investment (ROI) for a promotional offer.

Problem Statement

OpenAI is considering offering a 'One Month Free' promotion to dormant users to encourage them to subscribe to ChatGPT Plus. As a Data Scientist, you are tasked with designing an experiment to measure the causal impact of this promotion on long-term user value.

The Scenario

  • Goal: Increase the number of paid subscribers over a 6-month horizon.
  • Treatment: Users receive an email with a code for 1 month of free access.
  • Control: Users receive no offer.
  • Constraints: You have 10,000 users in the test pool. The data is prone to right-censoring, as users may cancel, churn, or renew at different times within the 6-month window.

Your Task

  1. Propose an experimental design to isolate the causal effect of the promotion. How do you handle selection bias and ensure the groups are comparable?
  2. Define the primary metric for the ROI of this promotion. How would you calculate the 'Incremental Profit' given the cost of the free month (server costs + lost revenue) versus the long-term subscription revenue?
  3. Propose a method to estimate the 6-month impact when you only have 2 months of post-promotion data.

Examples

  • Example Metric: If the Control group has a 5% conversion rate and the Treatment group has a 12% conversion rate, but 30% of Treatment users churn immediately after the free month, how do you adjust your ROI calculation?
  • Edge Cases: Consider users who were already planning to subscribe; how does the promotion affect their Lifetime Value (LTV)?

Hints

Hint 1

Consider using a Difference-in-Differences (DiD) framework or a Randomized Controlled Trial (RCT) with post-stratification to handle potential imbalances.

Hint 2

Think about using survival analysis techniques (like Kaplan-Meier curves or Cox Proportional Hazards) to account for users who have not yet reached the 6-month mark.

Hint 3

For ROI, differentiate between the 'Lift' in conversion and the 'Cannibalization' effect where users who would have paid anyway now get a free month.

Follow-up questions

  • How would you account for 'Switching Costs' or 'Lock-in' effects if the promo increases usage intensity during the free month?
  • If the treatment effect varies significantly by user segment (e.g., power users vs. casual users), how would you optimize the target audience for future rollouts?
  • What specific data quality issues or logging requirements would you prioritize to ensure the experiment is measuring actual revenue and not just trial sign-ups?