Glossary

Gradient checkpointing

Beginner

Saving memory while training by throwing away some in-between work and redoing it later when it’s needed.

Novice

A memory-saving trick: instead of keeping every intermediate result needed for the backward pass, keep only a few and recompute the rest. Training takes longer but needs much less memory.

Expert

Store activations only at layer boundaries and recompute each layer’s internals during backpropagation: about one extra forward pass (30–40% overhead in Megatron’s runs) for O(√n) or per-layer memory. On by default in TRL’s SFTConfig.