Onsite

Sharding Embeddings and Language-Model Outputs

MicrosoftSoftware Engineer · Onsite · Jan 2026

The author describes a Microsoft AI infrastructure virtual-onsite discussion centered on distributing large embedding tables and language-model output layers across GPUs. The interview compared splitting by vocabulary entries with splitting by hidden dimensions, asking how each choice affects memory use, communication, and connections to later computation. Follow-ups examined routing token lookup…

Shared by a candidate · reported Jan 2026. Interview formats change; use it as a guide, not a script.