Embedding
Beginner
A list of numbers that stands for a word piece. Pieces with similar meanings get similar lists.
Novice
A learned vector (a list of numbers) for each token, looked up in a table by the token’s ID. The model learns the values during training, so tokens used in similar ways end up with similar vectors.
Expert
A V×d matrix indexed by token ID; equivalent to multiplying a one-hot vector by the matrix. Often tied to the output projection.
See also: Positional encoding, Token.