Collective communication
Beginner
Group conversations between chips, where many chips swap data in one planned move.
Novice
Communication operations that involve a whole group of processors at once, such as all-reduce, all-gather, reduce-scatter and all-to-all, as opposed to one sender talking to one receiver.
Expert
Implemented by libraries such as NCCL as ring, tree or hierarchical schedules over the topology. Performance is usually reported as bus bandwidth, which normalizes the algorithm’s data volume to compare with link speed.
Explained in Why the network looks this way (Systems).
See also: All-reduce, All-to-all.