Design a scalable MoE pretraining pipeline
MetaMachine Learning Engineer · Onsite
Last updated
This question evaluates expertise in large-scale ML system design and distributed training for Mixture-of-Experts (MoE) models, covering competencies in model architecture, parallelism and communication strategies, memory and optimizer sharding, data curation and tokenization, monitoring, checkpointing, and fault tolerance; it is categorized under ML System Design and targets practical application with systems-level architectural reasoning rather than purely theoretical concepts. It is commonly asked to evaluate the ability to balance throughput, stability, and cost when scaling pretraining to hundreds of GPUs, reason about routing and capacity trade-offs, and design production-grade pipelines that can extend to larger clusters.
Design a large-scale Mixture-of-Experts (MoE) pretraining pipeline for a bilingual LLM trained on approximately 1T tokens across 256 A100-80GB GPUs. Specify the model architecture (number of experts, gating/routing strategy, top-k selection), parallelism plan (data/tensor/pipeline/expert), communication patterns (all-to-all), capacity factor and token dropping policy, load-balancing/auxiliary losses, memory and optimizer sharding (e.g., FSDP/ZeRO), checkpointing and fault tolerance, dataset curation and deduplication, tokenization, scheduling and hyperparameters, monitoring (tokens/sec, step time) and evaluation. Discuss key failure modes (router collapse, stragglers, expert overload) and concrete mitigations, and explain how you would scale the same system to 1,024 GPUs while controlling cost.
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.