Katkat
Back

Evaluate Metrics and Randomization for Onboarding Tutorial Change

ConfluentData Scientist · Onsite
MediumData ScientistOnsiteAnalytics & Experimentation

Last updated

Evaluate Metrics and Randomization for Onboarding Tutorial Change evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Scenario

Product team changed one specific step in Confluent’s user-onboarding tutorial and wants to evaluate whether the change improves the experience.

Question

Which primary and secondary metrics would you track that are highly specific to the modified tutorial step? 2. At which level would you randomize (user vs. account) and what covariates would you examine to verify comparable groups? 3. Which statistical test(s) would you use, how would you compute required sample size and expected runtime, and what alternative test would you prefer if the sample size turns out to be very small?

Hints

Think micro-conversion rates, time-to-complete, event drop-offs; discuss unit-of-analysis alignment and balance checks; consider t/Z tests, nonparametrics or Bayesian for small samples.