Tune classifier and compute key metrics

Problem Statement

You are building a fraud detection model for medical claims. Your binary classifier outputs a probability score p between 0 and 1 for each claim. You have two specific objectives:

1. Threshold Tuning: A False Positive (investigating a legitimate claim) costs $50 in administrative overhead. A False Negative (missing a fraudulent claim) costs the total value of the claim. Given a test set of claims with known labels and claim amounts, write a function to determine the optimal classification threshold that minimizes the total expected cost.

2. Hybrid Modeling: Once a claim is flagged as fraudulent, you need to estimate the 'Expected Loss' (the dollar amount of the fraud). How would you architect a system that uses the classifier's output to feed into a secondary regressor to predict the dollar amount of fraud?

Function Signature

def find_optimal_threshold(y_true, y_probs, claim_amounts, fp_cost=50):
    # y_true: List of 0s (legit) and 1s (fraud)
    # y_probs: List of model probability scores
    # claim_amounts: List of float values representing claim costs
    # Returns: float threshold
    pass

Constraints

• len(y_true) is up to 100,000.
• y_probs are floats in the range [0, 1].
• Minimize total cost = sum(cost_fp) + sum(cost_fn).

Example

• Labels: [0, 1], Probabilities: [0.2, 0.8], Amounts: [100, 200], FP Cost: 50.
• If threshold is 0.5:
  • Claim 1 (0.2) is predicted 0 (True Negative). Cost = 0.
  • Claim 2 (0.8) is predicted 1 (True Positive). Cost = 0.
  • Total = 0.

Edge Cases

• What if all claims are predicted as legitimate?
• How does the distribution of claim_amounts affect the sensitivity of your threshold choice?