Glossary

Layer normalization

Beginner

A step that evens out a list of numbers, so none grow too big or too small.

Novice

Rescales each token’s vector to have zero mean and unit spread, then applies a learned scale and shift. It keeps the numbers in a steady range as they pass through many layers, which makes training stable.

Expert

Per-token normalization over the feature dimension with gain γ and bias β (2d parameters). Pre-norm placement (before each sub-layer) is standard for deep stacks; RMSNorm drops the mean subtraction.