Glossary

Multi-head attention

Beginner

Running several attention steps side by side, each looking for a different kind of clue.

Novice

Attention split into several smaller “heads” that run in parallel, each with its own query, key and value projections, so different heads can track different relationships (meaning, the previous word, and so on).

Expert

d is split into h heads of size d/h; heads are computed independently and concatenated, then projected. The parameter count doesn’t depend on h.