Katkat
Back

Implement and explain positional encoding

Applied IntuitionMachine Learning Engineer · Technical Screen
MediumMachine Learning EngineerTechnical ScreenMachine Learning

Last updated

This question evaluates knowledge of positional encoding mechanisms for Transformer language models, covering embedding mathematics, tensor shapes and broadcasting, PyTorch implementation details, expected training and inference symptoms when positional information is omitted, and methods for empirical verification and ablation.

Implement positional encodings for a Transformer-based language model. Choose either sinusoidal or learned, show PyTorch code to compute and add them to token embeddings, explain the equations and tensor shapes, and integrate them into the model. Discuss expected symptoms if positional information is omitted and how you would verify the fix empirically.