Glossary

Feed-forward network (MLP)

Beginner

The part of each block where every word piece is worked on by itself, with no looking at others.

Novice

Two layers applied to each token separately: widen the vector (usually to 4× its size), apply a nonlinear function, and shrink it back. Most of a transformer’s parameters live here.

Expert

d→4d→dd \to 4d \to d with GELU (GPT-2) or a gated variant such as SwiGLU; 8d28d^2 weights per block, about two-thirds of the block.