Katkat
Back

Build a fraud detection model

ShopifyMachine Learning Engineer · Technical Screen
MediumMachine Learning EngineerTechnical ScreenMachine Learning

Last updated

Design an end-to-end machine learning system for real-time transaction fraud detection, covering label formulation under delayed feedback, feature design, threshold selection, and continuous monitoring.

Scenario

You are tasked with designing an automated transaction fraud detection model for an e-commerce platform processing hundreds of thousands of checkout orders per day across millions of independent merchant stores. When a buyer submits an order, the system must score the transaction in real time (within 100 milliseconds) and decide whether to allow, challenge (e.g., via 3D Secure / OTP), or decline the payment.

Data & Constraints

  • Scale: ~500 transactions per second (TPS) peak, ~20 million orders per month.
  • Base Rate: Historical confirmed fraud rate is approximately 0.15% of all transactions.
  • Label Delay: Chargebacks and merchant dispute notifications typically arrive 30 to 90 days after the actual transaction date.
  • Available Data: Current transaction metadata (amount, currency, shipping/billing address, IP, device fingerprint), buyer purchase history, merchant profile and risk tier, and payment card token attributes.

Tasks to Address

  1. Target Definition and Labeling Strategy

    • How do you construct the ground-truth training label given the 30–90 day label maturity window?
    • How do you handle unlabelled recent data and false negatives caused by un-reported fraud?

  2. Feature Engineering

    • Propose 4-5 high-signal features across buyer behavioral history, entity velocity, and cross-merchant relationships.
    • Explain how you compute low-latency streaming aggregations (e.g., velocity counts) at inference time without data leakage.

  3. Modeling and Class Imbalance

    • Choose an appropriate baseline model family and explain your trade-offs for latency vs. expressiveness.
    • How will you train the model given the extreme 0.15% class imbalance without introducing bias into predicted probabilities?

  4. Evaluation and Action Thresholding

    • Define your offline evaluation metrics (explain why raw accuracy or standard ROC-AUC are insufficient).
    • How do you set two operating decision thresholds: T_low (below which orders are allowed) and T_high (above which orders are blocked, with intermediate scores flagged for 3D Secure verification)?

  5. Feedback Loop and Monitoring

    • How do you monitor for concept drift, adversarial fraudster adaptation, and model degradation before chargeback labels arrive?

Hints

Hint 1

To train on recent data without severe label delay distortion, consider training only on data older than the maturity window or leveraging proxy labels such as early payment gateway declines.

Hint 2

For extreme class imbalance, avoid naive oversampling if you need calibrated probabilities; instead, consider cost-sensitive loss functions or downsampling with probability re-calibration.

Hint 3

Because chargebacks take months to arrive, proxy metrics like user contact rate, 3DS challenge failure rates, and feature distribution shift (PSI) must serve as early warning signals.

Follow-up questions

  • If you downsample non-fraud examples by a factor of 10 to speed up training, how do you mathematically adjust the predicted probabilities back to the true posterior scale?
  • How would you detect and adapt to a distributed card-testing attack spanning thousands of distinct merchant accounts within minutes?
  • How does the action taken on a transaction (declining vs. challenging) cause selection bias in your future training data, and how do you mitigate it?