Katkat
Back

Bias-Variance Tradeoff and Vanishing Gradients in Feedforward Networks

PinterestMachine Learning Engineer · Technical Screen
HardMachine Learning EngineerTechnical ScreenMachine Learning

Last updated

Tests theoretical and mathematical understanding of the bias-variance decomposition and the vanishing-gradient phenomenon during backpropagation in deep feedforward architectures.

Scenario

You are designing a deep feedforward neural network to predict Pin engagement (P(save | user, pin)) at Pinterest. During offline experiments with fully connected networks of varying depths (from 2 to 20 layers) using standard sigmoid activation functions, your team observes two distinct failure modes:

  1. Shallow networks (2-3 layers) converge rapidly but plateau at a high training loss and poor test metric.
  2. Very deep networks (15-20 layers) fail to train altogether, with weights in the early layers barely changing from their initial values.

Address both parts below to provide a theoretical analysis and practical diagnosis.

Part 1: Bias-Variance Tradeoff

  1. State the expected mean squared error decomposition for an estimator f_hat(x) predicting target y = f(x) + epsilon, where epsilon is zero-mean noise with variance sigma^2.
  2. Characterize the failure mode of the 2-layer network in terms of bias and variance.
  3. In modern deep learning, models with billions of parameters can achieve near-zero training error without suffering from high variance on unseen data (the "deep double descent" phenomenon). Explain conceptually how this differs from classical bias-variance tradeoff expectations.

Part 2: Vanishing Gradient Analysis

Consider an L-layer feedforward network where each layer l computes:

  • z^(l) = W^(l) * a^(l-1) + b^(l)
  • a^(l) = sigma(z^(l))

Here, a^(0) = x, and sigma is the element-wise sigmoid activation function sigma(z) = 1 / (1 + exp(-z)).

  1. Write out the expression for the gradient of the scalar loss L with respect to the first-layer weights W^(1), dLoss / dW^(1), using the chain rule.
  2. What is the maximum value of the first derivative of the sigmoid function, sigma'(z)? Explain mathematically why repeated multiplication by this term causes the magnitude of dLoss / dW^(1) to vanish exponentially as L increases.
  3. Compare two structural modifications that address this problem: replacing sigmoid with ReLU, and adding residual connections (a^(l) = sigma(z^(l)) + a^(l-1)). Explain how each modification prevents the gradient from vanishing.

Hints

Hint 1

Recall that the derivative of the sigmoid function can be written as `sigma(z) * (1 - sigma(z))`, which achieves its peak at `z = 0`.

Hint 2

During backpropagation, the Jacobian of each layer's transformation is chained together, scaling the initial loss gradient by the product of weight matrices and activation derivatives.

Hint 3

For residual connections, differentiate the identity path `a^(l) = F(a^(l-1)) + a^(l-1)` with respect to `a^(l-1)` to see how the gradient flows backward unattenuated.

Follow-up questions

  • How does the choice of weight initialization (such as Xavier/Glorot versus He initialization) mathematically interact with the vanishing or exploding gradient problem?
  • Under what data and capacity conditions do we observe the 'double descent' curve where test risk decreases again after interpolation?
  • If an activation function prevents vanishing gradients by having a derivative equal to 1 everywhere for positive values (like ReLU), what new failure modes can arise, and how are they mitigated?