Glossary

Scaling laws

Beginner

Rules that predict how much better a model gets when it is made bigger or given more practice.

Novice

Measured patterns showing that a language model’s loss falls smoothly and predictably as parameters, data and compute grow. They are used to plan how big a model to train and on how much data.

Expert

Power laws in N, D and C (Kaplan et al.); Hoffmann et al. found compute-optimal N and D grow about equally, roughly 20 tokens per parameter. They predict loss, not specific abilities.