Katkat
Back

Explain Overfitting and Transformer Attention

SnapchatMachine Learning Engineer · Technical Screen
MediumMachine Learning EngineerTechnical ScreenMachine Learning

Last updated

This question evaluates understanding of model generalization and regularization techniques alongside Transformer self-attention and positional encoding, assessing competencies in diagnosing overfitting, applying appropriate mitigation strategies, and interpreting attention mechanisms.

You are interviewing for a machine learning engineering role. Answer the following ML fundamentals questions clearly and compare different modeling settings.

  1. What is overfitting? How would you detect it using training and validation metrics?
  2. How would you reduce overfitting in a linear model?
  3. How would you reduce overfitting in a deep neural network?
  4. Explain the structure of Transformer self-attention. What are queries, keys, and values, and how are attention weights computed?
  5. Why does a Transformer need positional information? Describe at least two ways positional information can be added.