Activation
Beginner
The numbers flowing through a neural network: the input, and every in-between result one layer hands to the next.
Novice
A value produced by one layer of a neural network and consumed by the next. In a matrix multiplication the activations are one operand and the weights the other.
Expert
Activations change with every input while weights are fixed during inference, which is why weight-stationary designs pin the weights and stream activations. Their total size grows with batch and sequence length.