Relate coefficients under linear feature transformation

Linear Feature Transformation and Coefficient Invariance

Consider a standard multiple linear regression model: Y = X * beta + epsilon, where Y is an n x 1 vector of observations, X is an n x p design matrix (including an intercept column), beta is a p x 1 vector of coefficients, and epsilon is an n x 1 vector of i.i.d. Gaussian noise.

Suppose we perform a linear transformation on one of the features. Specifically, let the j-th predictor (where j > 1, i.e., not the intercept) be transformed from x_j to x'_j = a * x_j + b, where a is a non-zero scalar and b is a constant.

Tasks:

1. Derive how the coefficient beta_j for the transformed feature x'_j relates to the original coefficient beta_j in the new model Y = X' * beta' + epsilon.
2. Explain what happens to the intercept coefficient beta_0 due to this transformation.
3. If we standardize all features (i.e., a = 1/std(x_j) and b = -mean(x_j)/std(x_j)), how do the coefficients of the standardized model relate to the original ones, and why might this be preferred in practice?

Constraints:

• Assume X has full column rank.
• Assume OLS estimation is used for both models.

Example:

If Y = 2 + 3 * x1 and we transform x1 to x1' = 2 * x1 + 1:

• The new model becomes Y = 2 + 3 * ((x1' - 1) / 2) = (2 - 1.5) + 1.5 * x1' = 0.5 + 1.5 * x1'.
• Here, beta_0 changes from 2 to 0.5 and beta_1 changes from 3 to 1.5.