BF16 (bfloat16)
Beginner
A 16-bit number format that reaches numbers just as big and as tiny as the classic 32-bit format, but keeps fewer digits.
Novice
“Brain floating point”: 1 sign bit, 8 exponent bits and 7 mantissa bits. It is the top half of an FP32 number, so it has the same range (about to ) with roughly 2–3 significant decimal digits.
Expert
FP32 with the low 16 mantissa bits dropped (round-to-nearest-even on conversion). Same exponent and bias (127) as FP32, so no loss scaling is needed; precision , relative step . The standard training format on most accelerators, usually with FP32 accumulation.
Explained in Number formats (Architectures).
See also: FP16 (half precision), Mixed-precision training.