Katkat
Back

Define Churn and Design Onboarding Experiment

GustoData Scientist · Onsite
MediumData ScientistOnsiteAnalytics & Experimentation

Last updated

This question tests the ability to define business-relevant churn metrics for a SaaS platform and design a controlled experiment to optimize user onboarding.

The Problem

You are a Data Scientist at a payroll and HR platform. Management is concerned about early-stage user attrition. They have proposed a new 'Guided Setup' flow for new users, which replaces the existing 'Manual Setup'.

Part 1: Defining Churn

We define a customer as a business that has successfully completed their first payroll run. A 'churned' user is defined as a business that registered but failed to complete their first payroll run within 30 days.

  1. Identify three potential flaws in this definition and how you would adjust them to better capture user behavior.
  2. How would you handle users who pause their setup and return after 35 days? Does this count as churn and, if they eventually complete payroll, how is this handled in your data pipeline?

Part 2: Experiment Design

Design an A/B test to determine if the 'Guided Setup' reduces churn. Address the following:

  • Randomization Unit: Explain why you chose the user vs. the company/business level.
  • Metric Selection: Define your primary metric (the counter-metric) and one secondary metric to ensure the new flow doesn't degrade user experience.
  • Time Window: Given that the setup process can take several weeks, how do you determine when to stop the experiment to ensure statistical significance without waiting months?
  • External Factors: How do you control for seasonality (e.g., end-of-year tax season) impacting the results?

Hints

Hint 1

Consider the difference between a business that intended to sign up but abandoned the process versus one that just hasn't gotten around to running payroll yet.

Hint 2

For the experiment, think about the trade-off between the 'Primary Metric' (conversion) and 'Guardrail Metrics' (e.g., support tickets filed or feature adoption rate).

Hint 3

Regarding time windows, consider using survival analysis or cohort-based conversion rates rather than raw proportions.

Follow-up questions

  • If the 'Guided Setup' increases total signups but lowers the quality of businesses (lower lifetime value), how would you adjust the experiment's success criteria?
  • How would you account for network effects or marketing campaigns that might influence one group more than the other during the test period?
  • How do you ensure timezone consistency when calculating the '30-day' window for users across different geographic regions?