Glossary

Tokenizer

Beginner

The part that cuts text into small pieces and gives each piece a number, so a model can read it.

Novice

The program that turns text into a list of token IDs (integers from a fixed vocabulary) and back. It is trained once, before the model, and then never changes.

Expert

Usually byte-level BPE or a unigram model, with pre-tokenization rules (splitting on spaces, digits, punctuation). Its vocabulary sets the size of the embedding table and the output layer, and how many tokens a text costs.