Katkat
Back

Explain overfitting, underfitting, and regularization

PinterestMachine Learning Engineer · Technical Screen
HardMachine Learning EngineerTechnical ScreenMachine Learning

Last updated

This question tests the conceptual understanding of the bias-variance tradeoff and the practical application of regularization to balance model complexity and generalization error.

Conceptual Deep Dive: The Generalization Gap

You are training a high-capacity deep learning model to predict user engagement on Pinterest pins. After 50 epochs, you observe that the model achieves 99.9% accuracy on the training set but only 65% accuracy on the validation set.

  1. Diagnose the primary issue occurring here. Explain how this relates to the bias-variance tradeoff.

  2. Describe the mathematical intuition behind L1 (Lasso) and L2 (Ridge) regularization. Specifically, how do they modify the cost function, and what effect do they have on the weight vector magnitude?

  3. Suppose your model is now underfitting (high training error and high validation error). Propose three distinct architectural or procedural changes to mitigate this, explaining why they address underfitting specifically.

  4. How does the choice of regularization parameter (e.g., lambda) influence the model's position on the bias-variance spectrum?

Hints

Hint 1

Consider whether the large gap between training and validation performance suggests the model is capturing noise or failing to capture the underlying signal.

Hint 2

Think about the derivative of the penalty terms in L1 versus L2 and how that influences whether coefficients can become exactly zero.

Hint 3

For underfitting, focus on increasing model capacity or optimizing the training process to ensure the model has enough 'room' to learn the objective function.

Follow-up questions

  • How would you use 'Early Stopping' to manage overfitting, and what is its relationship to weight decay?
  • If you were using a tree-based model instead of a neural network, how would you address overfitting versus underfitting?
  • Can you explain the impact of increasing the size of the training dataset on the gap between training and validation performance?