Explain Mixture of Experts and Expert Parallelism
Mistral AISoftware Engineer · Technical Screen
Last updated
Explain sparse mixture-of-experts routing and expert parallelism, including token dispatch, weighted combination, load balancing, overflow policies, and communication costs.
Explain a mixture-of-experts transformer and how expert parallelism distributes its work. Trace what happens to a batch of tokens from routing through expert computation and combination.
Constraints & Assumptions
Use a sparse feed-forward MoE layer with top-k routing as the example. The number of experts, top-k value, capacity policy, and device placement are unspecified; state your choices when needed.
Clarifying Questions
Are all experts stored on one device or distributed? Can a token visit several experts? What happens when an expert receives more tokens than its capacity? Is the goal training throughput or serving latency?
Follow-up questions
- Why can a model have many total parameters but comparatively few active parameters per token? What does a load-balancing loss encourage? When can communication erase the expected compute benefit?
Solution walkthrough is for members
Get the full walkthrough with reference code and complexity analysis, plus Kitty Coach at 300 messages a day. Hints, the code runner and community answers stay free.