Lead structured response to accuracy incident
CVS HealthData Scientist · Technical ScreenLast updated
This question evaluates incident leadership, operational decision-making, and data-driven troubleshooting skills for a data scientist, focusing on triage, hypothesis formation, mitigation choice, cross-functional communication, and recurrence prevention.
You are the on-call lead for a payment-accuracy team. At 09:00 you learn that, since 00:00 today, the share of incorrect payments for users aged 18–24 in CA rose from a historical 2.0% to 3.1% across ~120k transactions; finance is escalating and wants a mitigation within 24 hours. You have access to feature logs, model scores, recent code/deployment diffs, and a sampling tool to pull raw cases for audit.
Describe, in a structured, time-boxed plan, exactly how you would: 1) Triage and quantify impact (owners, metrics, and acceptance criteria for 'all clear'), 2) Form testable hypotheses and design the minimum experiments/queries to isolate root cause (data slices, holdouts, backfills), 3) Decide on a safe mitigation (e.g., threshold change, feature hotfix, rule fallback), including rollback triggers and predicted side effects (precision/recall trade-offs), 4) Communicate status to execs and partner teams at specific times with concrete artifacts (dashboards/runbooks), and 5) Prevent recurrence (observability, guardrails, pre-deploy checks). Be explicit about risks you will accept vs. avoid and how you’d measure success by end of day.
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.