Katkat

Lead structured response to accuracy incident

CVS HealthData Scientist · Technical Screen

Last updated

This question evaluates incident leadership, operational decision-making, and data-driven troubleshooting skills for a data scientist, focusing on triage, hypothesis formation, mitigation choice, cross-functional communication, and recurrence prevention.

You are the on-call lead for a payment-accuracy team. At 09:00 you learn that, since 00:00 today, the share of incorrect payments for users aged 18–24 in CA rose from a historical 2.0% to 3.1% across ~120k transactions; finance is escalating and wants a mitigation within 24 hours. You have access to feature logs, model scores, recent code/deployment diffs, and a sampling tool to pull raw cases for audit.

Describe, in a structured, time-boxed plan, exactly how you would: 1) Triage and quantify impact (owners, metrics, and acceptance criteria for 'all clear'), 2) Form testable hypotheses and design the minimum experiments/queries to isolate root cause (data slices, holdouts, backfills), 3) Decide on a safe mitigation (e.g., threshold change, feature hotfix, rule fallback), including rollback triggers and predicted side effects (precision/recall trade-offs), 4) Communicate status to execs and partner teams at specific times with concrete artifacts (dashboards/runbooks), and 5) Prevent recurrence (observability, guardrails, pre-deploy checks). Be explicit about risks you will accept vs. avoid and how you’d measure success by end of day.