Katkat
Back

Explain bias–variance, overfitting, and vanishing gradients

PinterestMachine Learning Engineer · Technical Screen
MediumMachine Learning EngineerTechnical ScreenMachine Learning

Last updated

This question tests the conceptual understanding of the bias-variance tradeoff, methods for diagnosing and preventing overfitting, and the theoretical causes and architectural solutions for vanishing gradients in deep learning.

Part 1: Bias-Variance Tradeoff

Explain the bias-variance tradeoff in the context of supervised learning. How does increasing model complexity typically affect the bias and variance of a model's predictions, and what is the relationship between this tradeoff and the total error?

Part 2: Overfitting Diagnosis

You are training a deep neural network on a large image dataset. You observe that your training loss continues to decrease steadily, but the validation loss begins to increase after 50 epochs.

  1. Identify the phenomenon occurring and two specific techniques you would implement to mitigate it.
  2. Explain why simply collecting more training data is not always a sufficient solution for high-variance models.

Part 3: Vanishing Gradients

During the training of a very deep network with 100+ layers using the sigmoid activation function, the weights in the initial layers remain largely unchanged throughout the training process.

  1. Explain the mathematical intuition behind why the gradients vanish in this scenario.
  2. Propose two architectural modifications to solve this problem and explain why they are effective.

Hints

Hint 1

Consider how bias represents the error from erroneous assumptions, while variance represents sensitivity to small fluctuations in the training set.

Hint 2

Think about regularization techniques like dropout or weight decay that penalize complex patterns.

Hint 3

Recall the chain rule for backpropagation and how multiplying many values between 0 and 1 leads to an exponential decay of the gradient.

Follow-up questions

  • How does the choice of optimization algorithm (e.g., Adam vs. SGD) affect how a model handles high-variance scenarios?
  • If you were using ReLU activations instead of sigmoid, would you still encounter vanishing gradients? What about 'dying ReLU' problems?
  • How would you determine if your model has high bias or high variance if you only had access to training loss?