Katkat
Back

Explain parallelism and collectives in training

AmazonApplied Scientist · Onsite
MediumApplied ScientistOnsiteML System Design

Last updated

This question evaluates a candidate's competency in designing scalable distributed training systems, covering parallelism strategies (data, model/tensor, pipeline), communication collectives (all-reduce, all-gather, reduce-scatter, broadcast), and tensor-level layer partitioning such as column- and row-parallel splits.

Parallelism strategies and communication in large-scale training

You are designing a distributed training setup for very large neural networks that cannot fit on a single device.

Answer the following:

  1. Describe the main parallelism strategies used in large-scale training (for example, data parallelism, model/tensor parallelism, and pipeline parallelism). For each, explain how it works and its pros and cons.
  2. What are communication collectives (such as all-reduce, all-gather, reduce-scatter, and broadcast), and how are they used in distributed training?
  3. In tensor model parallelism, explain the idea of splitting linear layers into column-parallel and row-parallel parts. What is "alternating column and row parallelism" across layers, and why is it beneficial?