Explain Adam, AdamW, and Decoupled Weight Decay
Mistral AISoftware Engineer · Technical Screen
Last updated
Understand Adam moment estimates and bias correction, then compare gradient-based L2 regularization with AdamW’s decoupled weight decay and learning-rate interaction.
Explain Adam and AdamW. Why is adding an L2 penalty to the loss generally different from decoupled weight decay under Adam?
Constraints & Assumptions
Use standard first- and second-moment estimates, bias correction, and a scalar learning rate. State whether a parameter group is subject to decay; treatment of biases and normalization parameters is a separate choice.
Clarifying Questions
Is regularization added to the gradient or applied directly to parameters? What learning-rate schedule and epsilon convention are used? Which parameters should decay?
Follow-up questions
- When can L2 regularization and weight decay be equivalent? Why does the decay coefficient interact with the learning-rate schedule? Does AdamW remove the need to tune regularization?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.