Pretraining
Beginner
The long first practice, where a model reads huge amounts of text and learns to guess the next word.
Novice
The first and most expensive training stage: predicting the next token over a huge general corpus. It produces a base model that knows a lot of language and facts but doesn’t yet behave like an assistant.
Expert
Self-supervised next-token training on trillions of tokens; nearly all of the compute. Labels come from the text itself, so no annotation is needed.
See also: Fine-tuning, Scaling laws.