Katkat
Back

Diagnose Transformer training and inference bugs

OpenAIMachine Learning Engineer · Technical Screen
HardMachine Learning EngineerTechnical ScreenMachine Learning

Last updated

This question evaluates practical debugging skills for Transformer-based sequence models, targeting machine learning engineers who must diagnose intermittent shape/dtype errors and convergence failures simultaneously. It tests deep familiarity with the full training stack — data pipelines, attention masking, positional encoding, optimizer precision, and distributed setup — and the ability to reason about how a single root cause can manifest across multiple symptoms.

A Transformer-based sequence model intermittently throws shape/dtype mismatch errors and fails to converge after several thousand steps. Describe your end-to-end debugging approach. Include: how you validate tokenization, padding, and special tokens; verify attention masks (causal vs. bidirectional) and positional embeddings; add assertions and unit tests for tensor shapes and sequence lengths; pinpoint exploding/vanishing gradients (e.g., gradient norms, clipping, optimizer/betas, LR schedule, mixed precision); isolate data bugs (truncation, BOS/EOS handling, label shifting for LM objectives); diagnose loss NaNs and numerical instability; check checkpoint/resume logic and randomness/seed control; debug multi-GPU/DP/ZeRO issues (sync, grad scaling, AMP); profile throughput and memory (operator-level hotspots, OOM sources); and construct a minimal reproducible example with targeted logging to localize the fault. Provide concrete checks, metrics, and code-level instrumentation you would add.