Kernel fusion

Beginner

Doing several steps on data while it’s already on the chip, instead of sending it back to memory between each step.

Novice

Combining several operations into one program (kernel) so intermediate results stay in on-chip memory. It cuts memory traffic and raises arithmetic intensity.

Expert

The standard fix for chains of memory-bound ops. FlashAttention fuses $QK^T$ , softmax and $PV$ with tiling so the $S \times S$ score matrix never reaches HBM.

Explained in What the workload needs (Architectures).