Katkat
Back

Explain Layer Normalization in Transformers

AmazonApplied Scientist · Onsite
MediumApplied ScientistOnsiteMachine Learning

Last updated

This question evaluates a candidate's understanding of normalization techniques in deep learning—specifically Layer Normalization in Transformer blocks—covering the LayerNorm equation and the roles of gamma, beta, and epsilon, as well as contrasts with BatchNorm and RMSNorm, in the Machine Learning domain.

Explain Layer Normalization in Transformers. Where is it applied (pre-norm vs post-norm), and why does that choice affect gradient flow and training stability? Write the LayerNorm equation, discuss the roles of gamma/beta and epsilon, and contrast LayerNorm with BatchNorm and RMSNorm. Include guidance on initialization, placed-before vs placed-after residual connections, and implications for inference latency and memory.