Tokenizer
Beginner
The part that cuts text into small pieces and gives each piece a number, so a model can read it.
Novice
The program that turns text into a list of token IDs (integers from a fixed vocabulary) and back. It is trained once, before the model, and then never changes.
Expert
Usually byte-level BPE or a unigram model, with pre-tokenization rules (splitting on spaces, digits, punctuation). Its vocabulary sets the size of the embedding table and the output layer, and how many tokens a text costs.
See also: Byte-pair encoding (BPE), Token.