Explain Vanishing Gradients in Neural-Network Training

What is the vanishing-gradient problem, why can it occur when training a neural network, and what would you do to diagnose and reduce it? Explain how gradients propagate through repeated layers or time steps and distinguish very small backpropagated gradients from a model that simply has a low loss near a useful optimum.

You may illustrate the mechanism with a deep feed-forward network or a recurrent network, but state which setting you are using. No architecture or activation function is specified, so explain which proposed remedies depend on those choices. Discuss why a change that helps gradient propagation is not by itself evidence that the trained model generalizes better.