Katkat
Back

Design and evaluate a fraud detection strategy

PayPalData Scientist · Technical Screen
EasyData ScientistTechnical ScreenAnalytics & Experimentation

Last updated

This problem tests your ability to design a holistic fraud detection strategy by balancing technical model metrics with business-critical operational trade-offs.

The Scenario

You are a Data Scientist at a major payment processor. The fraud team is concerned about a spike in Account Takeover (ATO) attacks. ATO occurs when a bad actor gains unauthorized access to a legitimate user's account and performs fraudulent transactions.

We currently use a simple rule-based system (e.g., "block if IP is from a high-risk country") and are looking to deploy a machine learning-based risk scoring model. The model outputs a probability score between 0 and 1.

Your Task

  1. Metric Selection: Which metrics would you prioritize to evaluate the effectiveness of this model, and why? Contrast these with your metrics for the existing rule-based system.
  2. Threshold Strategy: How would you determine the optimal probability threshold for blocking transactions? Explain the trade-offs involved in setting this threshold too high versus too low.
  3. Operational Pipeline: How would you design a hybrid strategy that incorporates the ML model, the existing rule-based system, and manual review queues? Explain how you would route transactions to each stage.
  4. Evaluation: Suppose your model achieves high precision but low recall on ATO detection. What diagnostic steps would you take to investigate this, and what implications does this have for your operational strategy?

Hints

Hint 1

Consider the cost of a false positive (denying a legitimate customer) versus the cost of a false negative (allowing a fraudulent transaction).

Hint 2

Think about the 'Precision-Recall' trade-off and how business costs like customer churn or direct financial loss influence the choice of the Operating Point.

Hint 3

A hybrid approach often uses rules for high-confidence blocking and the model to flag 'gray area' transactions for manual investigation.

Follow-up questions

  • How would you account for concept drift in your fraud model over time as attackers change their methods?
  • If the model shows bias against certain geographic regions, how would you address fairness while maintaining detection performance?
  • What specific features (variables) would you engineer to differentiate between a legitimate user traveling abroad and an actual account takeover?