Adam / AdamW
Beginner
The rule that decides how big each nudge to the model’s numbers should be.
Novice
The optimizer most LLMs train with. It updates each weight using a running average of its gradients and of their squares, so each weight gets its own step size. AdamW also shrinks weights slightly each step (weight decay).
Expert
First and second moment estimates with bias correction; AdamW applies decoupled weight decay. Its two moments per parameter are the optimizer state that dominates training memory.
See also: Backpropagation, Optimizer state.