Katkat
Back

How would you define and use retention metrics?

MetaData Scientist · Technical Screen
EasyData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question evaluates a data scientist's competency in product analytics and experimentation by assessing understanding of retention metric definitions, cohorting strategies, short- versus long-term trade-offs, common measurement pitfalls, and A/B test design considerations.

Scenario

You are a Data Scientist supporting a consumer product (app or website). A PM asks you to “dive deep” on user retention and recommends tracking 7-day and 28-day retention.

Task

  1. Define retention clearly. Give at least three common retention definitions and explain how they differ:
  • N-day (classic/cohort) retention
  • Rolling retention (a.k.a. unbounded)
  • Return rate / weekly active retention (or another reasonable variant)
  1. Explain how you would compute and interpret 7-day vs 28-day retention:
  • What user cohorting would you use (e.g., signup/install week)?
  • What does each metric capture about user behavior?
  1. Discuss short-term vs long-term tradeoffs:
  • Give examples of product changes that might increase 7-day retention but harm 28-day retention (and vice versa).
  • Propose a metric hierarchy: primary, diagnostic, and guardrail metrics.

  1. Call out at least five pitfalls/edge cases when measuring retention and how you would address them (e.g., right-censoring, seasonality, re-installs, bots, changing definitions, missing events, timezone issues).

  2. If the PM wants to run an A/B test to improve retention, outline how you would design and evaluate it (unit of randomization, experiment duration, how to handle delayed effects, and any variance reduction or sequential testing considerations).