Glossary

Positional encoding

Beginner

Extra numbers that tell the model where each word sits in the sentence: first, second, third.

Novice

Information about each token’s position, added to its embedding. Without it, attention would treat the text as an unordered bag of tokens.

Expert

Sinusoidal (fixed), learned absolute (GPT-2), or relative schemes; RoPE rotates query and key vectors by a position-dependent angle so their dot product depends on the distance between tokens.