Glossary

Sequence packing

Beginner

Fitting several short examples into one long row, like packing a suitcase, so no space is wasted.

Novice

Placing several short training examples end to end in one fixed-length sequence, instead of padding each one, so less computation is wasted on padding.

Expert

Concatenating examples into max-length blocks (TRL: packing=True, best-fit decreasing by default) with attention kept within each example via position resets or a padding-free kernel. Speeds up SFT on short, varied data.