NF4 (4-bit NormalFloat)
Beginner
A way to store each of a model’s numbers in just 4 bits, chosen to fit the way those numbers spread out.
Novice
A 4-bit number format with 16 levels placed where a model’s weights usually fall, many near zero and few far out, so the rounding error is small for typical weights.
Expert
The 16 levels are quantiles of a standard normal (with an exact zero), applied per block of 64 weights scaled by the block’s absolute maximum. Double quantization then stores the scales in 8 bits per block of 256.
See also: QLoRA, Quantization.