Katkat
Back

How to Design Effective A/B Tests for Onboarding

NetflixData Scientist · Onsite
MediumData ScientistOnsiteAnalytics & Experimentation

Last updated

Evaluates A/B test design for a redesigned onboarding flow in a subscription app. Strong answers define activation, user-level randomization, primary and guardrail metrics, sample size, sequential monitoring, and decisions when activation improves but support burden rises.

Scenario

Product team plans to launch a redesigned onboarding flow and needs evidence it increases activation.

Question

Design an A/B test for the new onboarding. State hypothesis, unit of randomization, key metrics, guardrail metrics, and runtime calculation. If early results show uplift but increased support tickets, how would you decide whether to launch?

Hints

Address sample size, power, sequential checks, and balancing primary vs. secondary metrics.