Choose threshold under costs and uncertainty

Assume a deployment sends an incentive to predicted positives. Base rate π = 4% purchase without incentive. Benefit per true positive (incremental profit) = $50. Cost per false positive (incentive + email) = $1.00. Consider three candidate operating points from validation: A: TPR=0.70, FPR=0.12; B: TPR=0.55, FPR=0.05; C: TPR=0.80, FPR=0.20. (1) For a cohort of 100,000 users, compute expected incremental profit for A, B, C using E[profit] = 100000*(πTPR50 − (1−π)FPR1). Which threshold is best? (2) Provide a 95% CI for the chosen threshold’s profit using a delta method or a nonparametric bootstrap; specify what distributional pieces you resample and why. (3) Your model outputs probabilities; describe and compute two calibration diagnostics you would include (e.g., Brier score and reliability curve with ECE). (4) Outline a population drift test you would run monthly to ensure the chosen threshold remains optimal under shifting π.