Glossary

Reduce-scatter

Beginner

Everyone adds up their lists, but each chip ends up holding only its own slice of the total.

Novice

A collective that sums a buffer across all participants and leaves each participant with a different 1/p1/p slice of the result.

Expert

Moves (p−1)/p(p-1)/p of the buffer per participant at minimum. The first half of a bandwidth-optimal all-reduce, and the basis of sharded-optimizer (ZeRO/FSDP-style) training.

Explained in Scale-up fabrics (Systems).

See also: All-reduce, All-gather.

All 896 terms →