Sequence packing
Beginner
Fitting several short examples into one long row, like packing a suitcase, so no space is wasted.
Novice
Placing several short training examples end to end in one fixed-length sequence, instead of padding each one, so less computation is wasted on padding.
Expert
Concatenating examples into max-length blocks (TRL: packing=True, best-fit decreasing by default) with attention kept within each example via position resets or a padding-free kernel. Speeds up SFT on short, varied data.
See also: Context window.