Pipeline bubble
Beginner
Waiting time on an assembly line: at the start, later stations have nothing to do yet, and at the end, early ones have finished.
Novice
An empty slot moving through a pipeline, where a stage has no useful work. In a CPU it comes from a stall or a flushed instruction; in pipeline-parallel training it is the idle time while the pipeline fills at the start of a step and drains at the end, which shrinks when there are many more microbatches than stages.
Expert
In a CPU pipeline, a no-op inserted by a stall or a squash, costing one cycle of CPI per bubble. In pipeline-parallel training, of idle time per step with flushes, a fraction of ideal compute time; interleaving model chunks per device divides it by at times the point-to-point traffic.
Explained in Why the network looks this way (Systems).
See also: Pipelining, Pipeline parallelism, Microbatch.