Katkat
Back

Compare Batch Normalization and Layer Normalization

SnapchatMachine Learning Engineer · Onsite
HardMachine Learning EngineerOnsiteMachine Learning

Last updated

Compare batch and layer normalization by their axes, learned parameters, running statistics, batch-size effects, and training-to-inference debugging.

Compare batch normalization and layer normalization. Explain what each normalizes, how training and inference differ, and how you would diagnose a normalization-related training or serving problem.

Constraints & Assumptions

  • State the activation shape and normalization axes; the names alone do not specify an implementation's layout.
  • Use a minibatch of feature vectors as the baseline, then explain how convolutional or sequence inputs change the axes.
  • Distinguish trainable scale/shift parameters from statistics estimated from data.

Clarifying Questions to Ask

  • Is the model processing feature vectors, convolutional activations, or variable-length sequences?
  • What batch size is available during training and serving?
  • Are training and inference modes and saved normalization state handled consistently?

Hints

Hint 1

Write down the axes: Ask whether one example's normalized value depends on the other examples in the current batch.

Follow-up questions

  • Why can changing batch size affect batch normalization more directly than layer normalization?
  • What would you inspect if evaluation quality dropped when a trained model was switched into inference mode?