Katkat
Back

Designing a Bayesian Classifier for Case Analysis

AmazonApplied Scientist · Onsite
MediumApplied ScientistOnsiteMachine Learning

Last updated

Tests your ability to design a probabilistic Bayesian classification system under conditions of informative missingness, non-stationary class priors, and strict calibration requirements.

Scenario

At an e-commerce platform, the Seller Compliance team reviews incoming customer dispute cases to identify fraudulent claims (Y = 1) versus legitimate disputes (Y = 0). Only a small fraction of cases are reviewed by human investigators, and cases arrive with heterogeneous evidence signals extracted from buyer history, seller feedback, and transaction logs.

You are tasked with designing a Bayesian classifier to estimate the posterior probability P(Y = 1 | X) for each incoming case, which will be used directly to route high-risk cases to senior investigators and auto-resolve low-risk cases.

Data and Challenges

  1. Feature Modalities & Informative Missingness: Features X = (X_text, X_tabular, X_history) contain significant missing values. Crucially, missingness is not at random (MNAR); for example, legitimate occasional buyers often lack buyer-history metrics, whereas repeat fraudulent accounts systematically obfuscate specific fields.
  2. Dynamic Class Priors: The baseline rate of fraud fluctuates due to seasonal promotions and coordinated bad-actor campaigns (ranging from 1% to 12% across different weeks).
  3. Decision Thresholds and Calibration: Downstream workflows require well-calibrated probabilities rather than merely ranked scores, because automated actions carry asymmetric operational costs.

Tasks

Walk through the end-to-end modeling design:

  1. Generative vs. Discriminative Formulation: Specify how you would formulate P(Y = 1 | X). If choosing a generative Bayesian approach (e.g., Naive Bayes or a Bayesian graphical model) versus a discriminative model with Bayesian priors (e.g., Bayesian Logistic Regression / Bayesian Neural Network), justify your choice given the heterogeneous features.
  2. Handling Missingness: Formulate mathematically how you will incorporate informative missingness into your likelihood or feature representation without discarding unobserved attributes or assuming Missing Completely at Random (MCAR).
  3. Updating Dynamic Priors: Detail how you update class prior P(Y = 1) in near real-time without retraining the entire feature likelihood model P(X | Y).
  4. Calibration and Validation: Define the evaluation protocol to assess both discriminative power (e.g., PR-AUC) and probability calibration (e.g., Expected Calibration Error, reliability diagrams). Explain how you would correct calibration drift in production.

Hints

Hint 1

Consider introducing explicit binary missingness indicator variables to allow the model to learn the association between missing values and the target class.

Hint 2

Using Bayes' theorem, observe how the posterior log-odds can be decomposed into the log-prior ratio plus the log-likelihood ratio, making prior updates computationally trivial.

Hint 3

Evaluate calibration by binning predicted probabilities and comparing average predictions against observed empirical frequencies via reliability diagrams.

Follow-up questions

  • If certain feature values are conditionally dependent given the class label (violating naive conditional independence), how does that impact probability calibration, and how would you adjust for it?
  • How would you integrate expert investigator heuristics as informative conjugate priors over the model parameters?
  • Suppose human review labels are delayed by up to 14 days; how do you estimate the current week's prior `P(Y = 1)` when true outcomes are partially observed?