Microbatch
Beginner
A small slice of a batch of examples, sent through the model by itself.
Novice
A part of a training batch processed in one forward and backward pass. Gradients from several microbatches are added up before the weights are updated, and pipelines keep several microbatches in flight at once.
Expert
With global batch , data-parallel degree and microbatch size , each pipeline sees microbatches per step. sets the pipeline bubble, and sets kernel efficiency and activation memory.