Katkat
Back

Design fraud detection across channels with unknowns

AmazonData Scientist · Onsite
HardData ScientistOnsiteMachine Learning

Last updated

This question tests the ability to design an end-to-end fraud detection system that accounts for cold-start problems, cross-channel signals, and the trade-off between customer friction and loss mitigation.

Problem Statement

You are tasked with designing a fraud detection system for an e-commerce platform that operates across web, mobile app, and API-based third-party integrations. The system must detect fraudulent transactions in real-time. A major challenge is that fraud patterns evolve rapidly, and labeled data is often delayed by weeks (chargeback cycles). Furthermore, you must identify 'unknown' actors—users who have no prior history on the platform.

Your Task

Design an ML system that addresses the following:

  1. Feature Engineering: How do you capture cross-channel behavior (e.g., a user browsing on mobile but checking out via an API) and entity relationships (e.g., shared devices/IPs across accounts)?
  2. Model Architecture: Propose a model architecture that handles both supervised learning on historical labels and unsupervised/semi-supervised learning for zero-day fraud and unknown actors.
  3. Evaluation & Strategy: How do you define a cost-sensitive objective function that balances false positives (blocking legitimate users) against false negatives (fraudulent loss)?
  4. Operations: Describe a strategy for handling model drift and incorporating real-time feedback loops from fraud investigators.

Constraints

  • Latency: The inference must return a decision in under 200ms.
  • Data Quality: Label latency is high; however, 'inquiry' data (e.g., user clicked 'report fraud') is available within 24 hours.
  • Scalability: The system must handle 50,000 requests per second at peak.

Hints

Hint 1

Consider using graph-based features or embeddings to represent relationships between users, devices, and addresses, rather than relying solely on tabular user history.

Hint 2

For unknown actors, prioritize anomaly detection techniques like Isolation Forests or autoencoders, potentially refined by pre-trained LLMs to analyze transaction descriptions and behavioral metadata.

Hint 3

Formulate the loss function by assigning a higher penalty weight to high-value fraudulent transactions compared to the 'cost' of an unnecessary customer support intervention.

Follow-up questions

  • How would you integrate a LLM-assisted workflow to help human investigators review high-risk flags without introducing privacy or hallucination risks?
  • If we implement a 'step-up' authentication (e.g., SMS verification) for suspicious transactions, how would you measure the impact on conversion rates vs. fraud reduction?
  • How would you design a 'challenger' model pipeline to test new fraud detection strategies against the current production model without impacting live revenue?