Katkat
Back

Evaluate a free-trial A/B test

OpenAIData Scientist · Technical Screen
EasyData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question tests your ability to design an A/B test for a freemium product, interpret statistical results, and address potential biases like cannibalization or selection effects.

Scenario

OpenAI is testing a new 'Advanced Free-Trial' feature that grants free users access to a premium model for 3 days. Your goal is to determine if this trial increases long-term retention compared to the standard free model.

Task

  1. Define the primary success metric and a secondary 'guardrail' metric to ensure the test does not negatively impact the business.
  2. You observe that while the trial group has higher engagement during the 3 days, their engagement drops significantly on day 4 compared to the control group. How would you analyze this data to determine if the trial is actually beneficial?
  3. Propose a method to check for 'Selection Bias' if users were assigned to groups based on their account creation date rather than full randomization.

Constraints

  • Assume a large user base (millions of active users).
  • The primary goal is long-term retention (defined as active usage at day 30).

Example

  • If Group A (Control) has a 10% Day-30 retention and Group B (Trial) has an 8% Day-30 retention, despite higher Day-1 to Day-3 activity, explain the potential 'Novelty Effect' vs. 'Feature Cannibalization'.

Hints

Hint 1

Consider metrics that measure not just immediate usage, but the conversion rate to paid tiers or long-term daily active usage (DAU).

Hint 2

Think about how the 'cliff' on day 4 might indicate that users were simply burning through the trial and had no intent to stick around.

Hint 3

For selection bias, consider whether different cohorts (e.g., weekend sign-ups vs. weekday sign-ups) inherently behave differently, regardless of the treatment.

Follow-up questions

  • How would you adjust your analysis if you discovered that 20% of users in the trial group created multiple accounts to extend their trial period?
  • If the trial is statistically neutral on Day 30 but drastically increases infrastructure costs, what recommendation would you make to leadership?
  • How would you account for network effects if the premium feature involves sharing content that could influence the control group?