Byte-pair encoding (BPE)
Beginner
A way to pick word pieces: keep gluing together the two pieces that show up side by side most often.
Novice
A method for building a tokenizer’s vocabulary. Start from single characters (or bytes), then repeatedly merge the most frequent neighboring pair into a new token, until the vocabulary reaches its target size.
Expert
Greedy frequency-based merges learned on a corpus and replayed in order at encode time. Byte-level BPE starts from the 256 byte values, so any string can be encoded without an unknown token.
See also: Tokenizer.