Katkat
Back

Explain Neural-Network Regularization and Dropout

SnapchatMachine Learning Engineer · Onsite
HardMachine Learning EngineerOnsiteMachine Learning

Last updated

Explain L1, L2, dropout scaling, train/inference behavior, early stopping, and the difference between gradient penalties and decoupled weight decay.

Explain how regularization reduces overfitting in a neural network. Compare L1 and L2 penalties with dropout, including how dropout behaves during training and inference.

Constraints & Assumptions

  • Explain the mechanism rather than only naming techniques.
  • Use standard inverted dropout as the example and state the drop probability convention.
  • Distinguish a penalty added to the objective from changes to architecture, data, or the optimization process.

Clarifying Questions to Ask

  • Is the model overfitting, underfitting, or suffering from data leakage or distribution mismatch?
  • Which layers use dropout, and is the model in training or inference mode?
  • Does the optimizer implement an L2 gradient penalty or decoupled weight decay?

Hints

Hint 1

Preserve the expected activation: If a training activation is randomly set to zero, determine how surviving activations should be scaled so the conditional expected activation remains unchanged.

Follow-up questions

  • Why can too much dropout worsen both training and validation performance?
  • Why are an L2 penalty and decoupled weight decay not generally equivalent under adaptive optimizers?