In-network reduction
Beginner
The switch itself adds up the numbers passing through it. So the chips don’t have to send everything back and forth twice.
Novice
A switch feature in which the switch performs the additions of a reduction as data flows through it and sends the result back to every chip.
Expert
Each endpoint sends its buffer once and receives the result once, so the per-endpoint traffic is about instead of , roughly halving all-reduce time when bandwidth-bound. NVIDIA calls its version SHARP; the switch needs arithmetic units and buffer space.