Identify What Does Not Fix Vanishing Gradients
C3 AIData Scientist · Online Assessment
Last updated
Explain why raising the learning rate does not fix vanishing gradients and compare skip connections, ReLU, and batch normalization.
Identify What Does Not Fix Vanishing Gradients
Which change is not a reliable remedy for vanishing gradients in a deep neural network: adding skip connections, using an activation such as ReLU, raising the learning rate, or adding batch normalization? Explain the mechanism behind your choice and the limitations of the other interventions.
Hints
Hint 1
Separate signal from step size: The gradient is computed before the optimizer multiplies it by the learning rate.
Follow-up questions
- Can ReLU units still stop transmitting gradients?
- Why can residual connections help very deep models?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.