Katkat
Back

Analyze A/B Test Results for Subscription Conversion Rates

OpenAIData Scientist · Online Assessment
MediumData ScientistOnline AssessmentAnalytics & Experimentation

Last updated

Evaluates A/B test analysis for a limited-time free trial offer. Strong answers compute paid-conversion lift, account for trial starts and churn, test statistical significance, and make a launch decision using retained paid value, revenue, guardrails, and long-term holdouts.

Scenario

An A/B test offers free users a limited-time trial of the paid plan to see whether it increases paid subscriptions and reduces churn.

Question

Compute the signup (paid-conversion) rate for treatment vs. control and the percentage lift. 2. Test whether the lift is statistically significant (state your test, null/alt hypotheses, p-value or CI). 3. Calculate and compare cancel rates during the trial and after the first paid billing cycle. 4. Estimate net paid-subscriber change after 30 days and 60 days, incorporating both signups and cancels. 5. What additional metrics or user segments would you examine before recommending a full roll-out? 6. Summarize the experiment outcome and give a go / no-go recommendation with supporting numbers.

Hints

Standard A/B-testing framework: define metrics, check randomization, use proportion test or delta method, segment by tenure, bucketed time windows.