Glossary

Byte-pair encoding (BPE)

Beginner

A way to pick word pieces: keep gluing together the two pieces that show up side by side most often.

Novice

A method for building a tokenizer’s vocabulary. Start from single characters (or bytes), then repeatedly merge the most frequent neighboring pair into a new token, until the vocabulary reaches its target size.

Expert

Greedy frequency-based merges learned on a corpus and replayed in order at encode time. Byte-level BPE starts from the 256 byte values, so any string can be encoded without an unknown token.