Glossary

Backpropagation

Beginner

The way a model works out which of its numbers to nudge, and which way, after each guess.

Novice

The algorithm that computes, for every weight in a network, how much the loss would change if that weight changed (the gradient). It works backward from the loss through each layer using the chain rule.

Expert

Reverse-mode automatic differentiation: one backward pass costs about twice the forward pass in FLOPs, which is why training is about 6N FLOPs per token versus 2N for inference.