Katkat
Back

Implement and analyze custom attention

AnthropicSoftware Engineer · Onsite
HardSoftware EngineerOnsiteMachine Learning

Last updated

This question evaluates implementation and analysis skills for scaled dot-product attention, testing competencies in efficient tensorized PyTorch coding, numerical stability and mixed-precision handling, masking semantics (causal and padding), multi-head shape correctness, and unit-test validation including gradient and edge-case behavior.

Implement a custom attention module in PyTorch from scratch. Given Q, K, V tensors, compute attention scores, apply a scaling factor, masking, normalization, and produce the output. Ensure the code runs end-to-end with tests on shape correctness, gradient flow, and numerical stability. Explain why scaling (e.g., by 1/sqrt(d_k)) is used, when and why to apply normalization (e.g., softmax vs. layer normalization), and analyze the time and memory complexity. Propose optimizations for long sequences.