Scaling laws
Beginner
Rules that predict how much better a model gets when it is made bigger or given more practice.
Novice
Measured patterns showing that a language model’s loss falls smoothly and predictably as parameters, data and compute grow. They are used to plan how big a model to train and on how much data.
Expert
Power laws in N, D and C (Kaplan et al.); Hoffmann et al. found compute-optimal N and D grow about equally, roughly 20 tokens per parameter. They predict loss, not specific abilities.
See also: Pretraining.