Collective operation
Beginner
A team job where every chip takes part at once, like everyone adding their numbers together and all getting the total.
Novice
A communication step in which all members of a group participate together: broadcast, reduce, all-reduce, all-gather, reduce-scatter and all-to-all. Libraries such as MPI and NCCL provide them.
Expert
A group-wide communication pattern with a defined result (MPI and NCCL semantics). Its cost depends on the algorithm (ring, tree, recursive halving/doubling, direct) mapped onto the physical topology; libraries switch algorithms by message size and group size.
Explained in Scale-up fabrics (Systems).
See also: All-reduce, Reduce-scatter, All-gather.