FP16 (half precision): definition

Beginner

The standard 16-bit number format. It keeps more digits than BF16 but can’t hold very big or very tiny numbers.

Novice

IEEE half precision: 1 sign bit, 5 exponent bits, 10 mantissa bits. Its largest value is 65,504 and the smallest non-zero value is about $6 \times 10^{-8}$ , so small gradients can vanish without help.

Expert

IEEE binary16: bias 15, $p = 11$ , max 65,504, min normal $2^{-14}$ , min subnormal $2^{-24}$ . Three more mantissa bits than BF16 but far less range (about 30 binades of normal range against 254); training with it needs loss scaling and FP32 master weights.