Glossary

Pretraining

Beginner

The long first practice, where a model reads huge amounts of text and learns to guess the next word.

Novice

The first and most expensive training stage: predicting the next token over a huge general corpus. It produces a base model that knows a lot of language and facts but doesn’t yet behave like an assistant.

Expert

Self-supervised next-token training on trillions of tokens; nearly all of the compute. Labels come from the text itself, so no annotation is needed.