Katkat
Back

Explain Transformer Encoder and Decoder Behavior

Point72Quantitative Researcher · Technical Screen
MediumQuantitative ResearcherTechnical ScreenMachine Learning

Last updated

This question evaluates knowledge of Transformer architecture and generative language modeling, specifically the encoder versus decoder roles, attention patterns and causal masking, plus mechanisms behind stochastic text generation such as sampling and temperature.

Answer the following Transformer fundamentals questions in a machine learning interview:

  1. What are the main differences between a Transformer encoder and a Transformer decoder? Your answer should discuss attention patterns and the role of a causal mask.
  2. For the same input prompt, why can an autoregressive decoder-based language model produce different outputs on different runs? Discuss sampling, temperature, and when generation would be deterministic.