Stochastic rounding: definition

Beginner

Rounding up or down at random, more often toward the closer value, so tiny changes add up correctly on average.

Novice

A rounding mode where a value between two representable numbers rounds up with probability equal to how close it is to the upper one. On average the rounding error is zero, so many small updates are not all lost.

Expert

Round $x$ to $\lfloor x \rfloor$ or $\lceil x \rceil$ with $P(\text{up}) = (x - \lfloor x \rfloor)/\mathrm{ulp}$ ; unbiased ( $E[\mathrm{round}(x)] = x$ ). Needs a random source per conversion. Used for low-precision weight updates and gradient quantization (for example in FP4 training) where round-to-nearest would systematically drop small increments.