Katkat
Back

Build a real-time ATO model

PayPalData Scientist · Technical Screen
HardData ScientistTechnical ScreenMachine Learning

Last updated

This question assesses the ability to design an end-to-end, low-latency machine learning pipeline for detecting account takeover (ATO) in payment transactions, focusing on label noise, feature engineering, and system deployment constraints.

Scenario

You are tasked with building a real-time Account Takeover (ATO) detection system for a payment processing platform. Every authorization request must be scored by a model within 50ms.

The Problem

  1. Labeling Challenges: Fraud labels are often delayed (e.g., a user reports unauthorized activity 3–5 days later). Propose a strategy to handle this 'label latency' during model training.
  2. Feature Engineering: Given the high-velocity nature of payments, define three categories of features that would be effective for ATO detection. Explain how you would compute these at sub-millisecond latency.
  3. Model Architecture: Describe the model architecture and the trade-offs between a batch-trained Gradient Boosted Decision Tree (GBDT) and an online learning approach (e.g., Online Logistic Regression or streaming neural nets).
  4. System Integration: How would you handle model calibration in an environment where the 'true' fraud rate is unknown and the cost of false positives (declining a legitimate user) is significantly different from the cost of false negatives (allowing a fraudulent transaction)?

Evaluation

Provide a high-level architecture diagram (described in text) and justify your choices for the feature store, model inference service, and the feedback loop to update the model with delayed labels.

Hints

Hint 1

Consider using an 'observation window' or 'waiting period' for labels, and discuss how to mitigate sample bias during the wait time.

Hint 2

For low-latency feature computation, explore the use of an in-memory key-value store (e.g., Redis) to track rolling aggregates of transaction velocity and device fingerprints.

Hint 3

Address the cold-start problem and how you might use a 'reject-inference' technique to compensate for the fact that you only see labels for transactions that were not initially blocked by a heuristic rule.

Follow-up questions

  • How would you measure the performance of this system in A/B testing without impacting the user experience for legitimate customers?
  • If you detect a sudden shift in the feature distribution (e.g., a new botnet using spoofed hardware identifiers), what steps would you take to retrain or adjust the model?
  • How does your system handle 'concept drift' versus 'data drift' specifically in the context of evolving fraud tactics?