Residual connection
Beginner
A shortcut that adds a step’s result on top of what went in, instead of replacing it.
Novice
A connection that adds a layer’s input to its output. Each layer then only has to learn a correction, and the signal (and its gradient) can travel straight through a very deep network.
Expert
. In a transformer the sum of all sub-layer outputs forms the residual stream; gradients reach early layers through the identity path.
See also: Layer normalization.