Causal mask
Beginner
A rule that stops each word from peeking at the words after it, like covering the rest of a page.
Novice
In a text-generating model, each position may attend only to itself and earlier positions. The mask hides later tokens, so training matches generation, where the future isn’t written yet.
Expert
An additive mask of −∞ above the diagonal of the score matrix before softmax. It lets one forward pass train on every position of a sequence at once.
See also: Attention.