Compare RMSNorm and LayerNorm in Transformer Blocks
Mistral AISoftware Engineer · Technical Screen
Last updated
Compare LayerNorm and RMSNorm equations, centering, shift behavior, hidden-feature axes, numerical precision, and the tradeoffs behind simpler normalization.
Compare LayerNorm and RMSNorm in a transformer block. Write their equations, explain the effect of omitting mean subtraction, and discuss why RMSNorm can be an attractive design choice.
Constraints & Assumptions
Normalize over the hidden features of one token. Use the common affine LayerNorm and scale-only RMSNorm definitions; implementations can vary in bias, epsilon placement, and precision.
Clarifying Questions
Which axes are normalized? Is a learned bias included? Is normalization before or after the sublayer? What numerical precision is used for reductions?
Follow-up questions
- What happens if the same constant is added to every feature? How do epsilon and low-precision arithmetic affect the result? Does RMSNorm guarantee a zero-mean output?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.