Glossary

NF4 (4-bit NormalFloat)

Beginner

A way to store each of a model’s numbers in just 4 bits, chosen to fit the way those numbers spread out.

Novice

A 4-bit number format with 16 levels placed where a model’s weights usually fall, many near zero and few far out, so the rounding error is small for typical weights.

Expert

The 16 levels are quantiles of a standard normal (with an exact zero), applied per block of 64 weights scaled by the block’s absolute maximum. Double quantization then stores the scales in 8 bits per block of 256.