Glossary

Embedding

Beginner

A list of numbers that stands for a word piece. Pieces with similar meanings get similar lists.

Novice

A learned vector (a list of numbers) for each token, looked up in a table by the token’s ID. The model learns the values during training, so tokens used in similar ways end up with similar vectors.

Expert

A V×d matrix indexed by token ID; equivalent to multiplying a one-hot vector by the matrix. Often tied to the output projection.