Katkat
Back

Identify and validate risky assumptions

Capital OneData Scientist · Technical Screen
MediumData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This problem tests your ability to deconstruct a business proposal, identify underlying behavioral and statistical assumptions, and design a validation strategy to mitigate product risk.

Scenario

A digital bank intends to launch a 'Smart Savings Round-Up' feature. When a user makes a purchase using their debit card, the transaction is rounded up to the nearest dollar, and the difference is automatically moved into a high-yield savings account. The product team assumes this will increase total deposits by 15% and improve long-term customer retention by 10%.

As the lead Data Scientist, you are tasked with auditing this proposal before the pilot launch.

Your Task

  1. Identify three critical, risky assumptions the product team has made regarding customer behavior or market response.
  2. For each assumption, propose a specific way to validate it using either historical data, a survey, or a small-scale A/B test before committing to a full launch.
  3. Discuss potential unintended consequences (e.g., impact on core checking account balances) and how you would monitor them.

Hints

Hint 1

Consider the difference between what customers say they will do (survey) and what they actually do when money is taken from their checking account.

Hint 2

Think about the 'substitution effect'—could this feature cause users to move money out of their existing savings accounts rather than creating 'new' deposits?

Hint 3

Evaluate whether the 15% deposit increase is dependent on specific user segments, such as high-frequency spenders versus low-income users.

Follow-up questions

  • How would you handle the trade-off if the round-up feature succeeds in boosting savings but leads to an increase in overdraft fees for lower-income users?
  • If you were to design an A/B test for this feature, what would be the primary metric, and what guardrail metrics would you set to ensure no negative impact on overall account health?
  • How would you adjust your estimation of 'incremental' deposits if users simply transfer money from an external account into their checking account just to have it rounded up?