Katkat
Back

Explain learning-rate fluctuation and vanishing gradients

PinterestMachine Learning Engineer · Technical Screen
EasyMachine Learning EngineerTechnical ScreenMachine Learning

Last updated

This question evaluates a candidate's understanding of optimization dynamics (effects of learning rate on training stability) and backpropagation-related issues (vanishing gradients) in deep neural networks, assessing competencies in training behavior and gradient propagation within the Machine Learning domain.

ML Fundamentals

Answer the following conceptual questions:

  1. Learning rate vs. training stability: Why can training metrics (loss/accuracy) fluctuate or oscillate when the learning rate is too large? What happens when it is too small?

  2. Vanishing gradients in fully connected networks: In a deep fully connected network trained with backpropagation, are vanishing gradients more likely to affect layers closer to the input or closer to the output? Explain why, and name common mitigations.