Katkat
Back

Choose and compute recommender evaluation metrics

MetaData Scientist · HR Screen
HardData ScientistHR ScreenMachine Learning

Last updated

This question evaluates competency in offline recommender-system evaluation, propensity-weighted policy comparison, classification metric interpretation, probability calibration, model selection trade-offs, and handling data leakage and network interference.

You’re building a restaurant recommender with a logistic regression scorer p(y=1|x) for whether a user engages with a recommendation. A/B tests and user surveys are unavailable for now.

  1. Offline evaluation design: Propose a defensible offline protocol to compare two models M0 and M1 without A/B (e.g., counterfactual evaluation with inverse propensity scores from historical logging policy). Specify the exact metric(s) you’ll compute (e.g., PR-AUC, Precision@K, calibrated Brier score), how you’ll avoid leakage, and how you’ll choose K. Name one pitfall of IPS when propensities are small and a mitigation.
  2. Thresholded metrics: Suppose on a holdout set of 1,000 recommendations at threshold 0.7 you observe: TP=120, FP=30, TN=820, FN=30. Compute precision, recall, specificity, F1, accuracy. Explain why accuracy can be misleading here and which metric aligns best if the goal is “every shown item should be relevant.”
  3. Calibration: Describe how you would check and fix probability calibration (e.g., reliability diagrams, Platt scaling vs. isotonic). Why does good calibration matter when setting business rules like “only show if score ≥ τ”?
  4. Model choice: Justify logistic regression over more complex models for this case; name two failure modes (e.g., feature multicollinearity, class imbalance) and concrete fixes.
  5. Network effects: If friend activity features introduce interference, what offline split strategy reduces leakage (e.g., time-based, user-disjoint, or graph-clustered splits)? State the trade-offs.