Prevent Vanishing Gradients in Deep Networks

Scenario

At Pinterest, you are designing a deep user-engagement sequence and ranking model. To capture complex cross-feature interactions and multi-year pin interaction histories, the team tests scaling a feed-forward ranking tower from 6 layers to 40 fully connected layers, alongside testing an unrolled recurrent sequence model (e.g., standard RNN) over user click streams of length T = 100.

During training of both models with standard initialization and sigmoid activations, training loss plateaus almost immediately. The gradients of the loss with respect to early layer weights are orders of magnitude smaller than those of the final layers.

Tasks

Evaluate the failure modes and design an end-to-end strategy to stabilize gradient propagation:

1. Mathematical Derivation & Intuition: Write out the chain rule formulation for the gradient of the loss L with respect to weight matrix W_1 in an L-layer feed-forward network h_l = f(W_l * h_{l-1} + b_l). Show explicitly how the spectral norm or magnitude of Jacobian matrices J_l = d(h_l) / d(h_{l-1}) causes exponential decay (vanishing) or growth (exploding) of gradients as L increases.
2. Mitigation Analysis:
   • Activations & Initialization: Contrast Sigmoid, ReLU, and Leaky ReLU / GELU. Explain the mathematical condition that Xavier/Glorot and He/Kaiming initializations enforce on the variance of activations and gradients, and why pairing matters.
   • Residual Connections: Formulate the forward pass h_l = h_{l-1} + F(h_{l-1}) and derive d(h_L) / d(h_l). Explain why this creates an "unobstructed gradient highway" that directly prevents vanishing gradients.
   • Normalization: Compare LayerNorm and BatchNorm. Explain mechanistically how normalizing layer inputs stabilizes the backward pass.
3. Optimization vs. Regularization: Identify which of the above techniques primarily assist optimization (loss minimization efficiency) versus regularization (generalization to unseen data), and where trade-offs emerge.
4. Feed-Forward vs. Recurrent Dynamics: Contrast how vanishing gradients manifest in a 40-layer feed-forward network versus an unrolled RNN across 100 time steps. Explain why simple weight initialization is often insufficient for vanilla RNNs and how gated architectures (LSTM/GRU) or modern Transformers solve the problem.