Katkat
Back

How to debug an apparent D14 retention drop

InstacartData Scientist · Technical Screen
EasyData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This question evaluates competency in cohort-based retention analysis, understanding metric maturity and right-censoring, performing data quality and definition checks, and localizing real declines across segments, releases, or funnel changes.

Scenario

A dashboard shows D14 retention (users retained on day 14 after signup/first activity). In the last week, the chart shows a sharp decline.

Assume retention is computed as a cohort metric:

  • Users are assigned to a cohort by first activity date.
  • D14 retention for a cohort is measured 14 days later.

Task

Explain how you would determine whether this decline reflects a real product issue or a false alarm.

Requirements

Include:

  • How metric “maturity” / delayed observation can create misleading recent-week dips.
  • What plots/tables you would inspect (cohort table, maturity curve, right-censoring).
  • Data quality checks and definition checks.
  • If it’s real, how you would localize the cause (segments, releases, funnel changes).