Katkat
Back

Explain How L1 and L2 Regularization Change Linear-Model Coefficients

Point72Data Scientist · Onsite
MediumData ScientistOnsiteMachine Learning

Last updated

This question tests the conceptual understanding of how L1 and L2 penalties influence the optimization objective and the resulting structure of the weight vector in a linear regression model.

The Problem

You are building a linear regression model to predict stock returns based on a high-dimensional set of 500 potential financial indicators. You are comparing Ordinary Least Squares (OLS) with Lasso (L1) and Ridge (L2) regularization.

Explain the mathematical and geometric impact of L1 and L2 penalties on the coefficient estimates. Specifically, address the following:

  1. Contrast the penalty terms added to the OLS cost function: sum((y - Xw)^2) + lambda * ||w||_1 versus sum((y - Xw)^2) + lambda * ||w||_2^2.
  2. Explain why L1 regularization often results in sparse coefficient vectors (many weights equal to exactly zero), whereas L2 regularization does not.
  3. Discuss the implications for feature selection and model interpretability in the context of the 500 indicators.

Theoretical Framework

  • Assume w is the vector of coefficients.
  • Define the constraint regions for the budget t where ||w||_1 <= t and ||w||_2^2 <= t.
  • Discuss the gradient of the penalty terms at w = 0.

Hints

Hint 1

Visualize the contour lines of the OLS loss function interacting with the diamond-shaped constraint region of L1 and the circular constraint region of L2.

Hint 2

Consider the derivative of the absolute value function at zero versus the derivative of the squared value function at zero.

Hint 3

Think about the effect of the 'corner' of the L1 diamond on the optimization path.

Follow-up questions

  • How would you determine the optimal value of the regularization parameter (lambda) in a production setting?
  • What happens if there is high multicollinearity among your 500 indicators when using Ridge versus Lasso?
  • How would you implement Elastic Net, and what specific problem does it solve that Lasso or Ridge alone cannot?