Katkat
Back

Explain Transformer Positional Encoding

CadenceMachine Learning Engineer · Technical Screen
MediumMachine Learning EngineerTechnical ScreenMachine Learning

Last updated

This question tests your conceptual and technical understanding of why Transformers require positional encodings and the mathematical mechanics of different encoding strategies.

The Role of Positional Encoding in Transformers

Unlike Recurrent Neural Networks (RNNs) or Convolutional Neural Networks (CNNs), the Transformer architecture processes input sequences in parallel using self-attention mechanisms. By default, the self-attention operation is permutation-invariant, meaning it treats an input sequence as a "bag of words" and cannot intrinsically distinguish between different token orderings.

Your Task

  1. Explain why standard self-attention is permutation-invariant and why this is problematic for sequence-to-sequence tasks (e.g., machine translation).
  2. Contrast the two primary approaches to positional encoding:
    • Sinusoidal Encoding: The original implementation using sine and cosine functions of different frequencies.
    • Learned Embeddings: The approach where positional vectors are treated as trainable model parameters.
  3. Discuss the concept of relative positional information. Why is it often considered superior for handling sequence lengths longer than those seen during training?

Mathematical Context

In the sinusoidal approach, the encoding for position pos at dimension i is defined as:

  • PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
  • PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

Briefly explain why this specific geometric progression allows the model to attend to relative positions.

Hints

Hint 1

Consider that the attention score is calculated as a dot product of Query and Key vectors, which does not account for the absolute index of a word.

Hint 2

Think about the property that any sinusoidal function at position pos+k can be represented as a linear combination of the values at pos and k.

Hint 3

Consider the memory constraints of learned embeddings when deploying a model on sequences significantly longer than the fixed size allocated during training.

Follow-up questions

  • How do Rotary Positional Embeddings (RoPE) differ from the original sinusoidal approach, and why have they become the standard in modern LLMs like Llama?
  • If you were tasked with building a model for images, how would you adapt positional encoding to represent a 2D grid instead of a 1D sequence?
  • What are the trade-offs in computational overhead when using dynamic relative bias embeddings versus static absolute positional embeddings?