Katkat
Back

Compute and interpret quantile loss vs RMSE

AmazonData Scientist · Technical Screen
MediumData ScientistTechnical ScreenStatistics & Math

Last updated

This question tests the ability to distinguish between point-prediction error metrics and quantile loss, and the practical implications of choosing an evaluation metric in the context of asymmetric business costs.

The Scenario

You are working on an inventory management system for a retail chain. Your model predicts the daily demand for a specific product. You have two options for your model's output:

  1. A point estimate (a single number representing the expected demand).
  2. A probabilistic forecast (quantiles, such as the 10th, 50th, and 90th percentiles of the demand distribution).

Your Task

  1. Define the Quantile (Pinball) Loss function for a quantile tau and a predicted value y_hat given an actual observation y: L(y, y_hat, tau) = (y - y_hat) * tau if y >= y_hat L(y, y_hat, tau) = (y_hat - y) * (1 - tau) if y < y_hat

  2. Explain why a model trained to minimize RMSE (Root Mean Squared Error) typically converges to the mean of the conditional distribution, whereas a model trained to minimize Quantile Loss at tau=0.5 converges to the median.

  3. Suppose the cost of a stockout (under-prediction) is 5 times higher than the cost of overstocking (over-prediction). Explain how you would utilize the Quantile Loss function to determine the optimal quantile to target for your inventory level.

Hints

Hint 1

Consider the derivative of the loss function with respect to the prediction and how it relates to the probability density function of the target variable.

Hint 2

Think about the 'asymmetric cost' as a weighting mechanism where the penalty for errors changes depending on whether the prediction is too high or too low.

Hint 3

The optimal quantile is often related to the critical ratio in the Newsvendor Model, which balances the costs of underage and overage.

Follow-up questions

  • How does the empirical coverage of your prediction intervals change if your model is biased?
  • If you have a heteroskedastic demand distribution, how would the quantile regression approach perform compared to a simple global RMSE minimization?
  • What are the trade-offs of using Quantile Loss versus a proper scoring rule like Continuous Ranked Probability Score (CRPS)?