All-reduce
Beginner
Every chip starts with its own list of numbers; at the end, every chip holds the sum of all the lists.
Novice
A collective that combines (usually sums) a buffer from every participant element by element and leaves the full result on every participant. Training uses it to add up gradients across chips.
Expert
Semantically reduce + broadcast; efficiently implemented as reduce-scatter followed by all-gather, which moves of the buffer per participant, the bandwidth lower bound for algorithms that don’t reduce inside the network.