Katkat
Back

Explain BatchNorm, optimizers, and L1/L2

SnapchatMachine Learning Engineer · Onsite
MediumMachine Learning EngineerOnsiteMachine Learning

Last updated

This question tests your conceptual understanding of how normalization, optimization strategies, and regularization techniques influence training stability, convergence, and model sparsity.

The Problem

Explain how the following three pillars of deep learning affect training dynamics and model performance. Provide a concise explanation for each section:

1. Batch Normalization (BN)

  • Describe the motivation for using BN in deep neural networks. How does BN behave differently during training versus inference, and why is this distinction necessary?
  • What is the internal covariate shift, and how does BN mitigate the vanishing/exploding gradient problem?

2. Optimization Dynamics

  • Compare and contrast SGD with Momentum, RMSProp, and Adam. In what specific loss landscape scenarios would you choose Adam over SGD with Momentum?
  • Explain the role of the "second moment" term in adaptive optimizers like Adam. What are the potential downsides of using adaptive learning rates in terms of generalization?

3. Regularization (L1 vs. L2)

  • Mathematically or geometrically explain why L1 regularization often results in sparse weight vectors (many weights becoming exactly zero), whereas L2 regularization results in smaller, non-zero weight values.
  • If you are building a model where feature selection is a business requirement, which regularization method would you prefer and why?

Hints

Hint 1

For Batch Norm, consider what happens to the mean and variance statistics when the model is in evaluation mode versus when it is processing mini-batches.

Hint 2

For optimizers, consider the shape of the error surface (e.g., ravines vs. plateaus) and how adaptive methods normalize the gradient updates based on past information.

Hint 3

For regularization, visualize the constraint regions (a diamond shape vs. a hypersphere) and how the loss function contour levels intersect with them.

Follow-up questions

  • If you notice your training loss is fluctuating significantly even with a small learning rate, which of these components would you investigate first?
  • How does Batch Normalization interact with dropout when they are used in the same layer architecture?
  • Can you describe a scenario where L2 regularization might actually harm the performance of a model?