Number formats

Given that a network is mostly matrix multiplication (What the workload needs), the number format sets three things at once: bytes per operand, which moves the arithmetic intensity of every kernel; multiplier area and energy, which grow roughly with the square of significand width⁵⁶; and the numerical error the model has to absorb. The history is a steady walk down that ladder:

• FP32 → 16-bit with mixed precision: 16-bit operands, FP32 accumulation, FP32 master weights, and loss scaling for FP16.¹ BF16 removed the need for loss scaling by keeping FP32’s exponent.⁷
• 16 → 8-bit floats: E4M3 and E5M2, standardized by OCP, used with per-tensor scale factors.²³
• Per-tensor → per-block scaling: the OCP MX specification (MXFP8, MXFP6, MXFP4, MXINT8), because a single scale is not enough below 8 bits.⁴⁸

In parallel, inference has long used integer quantization with INT8 operands and INT32 accumulators.⁹ This chapter goes through the encodings bit by bit, the scaling schemes that make narrow formats usable, the error each introduces, accumulation precision, and the hardware cost model that motivates all of it. The algorithms section derives the error models.

Encoding

An ExMy format has 1 sign bit, $x$ exponent bits and $y$ mantissa (trailing significand) bits.⁴ With bias $b$, a normal encoding ($E > 0$) has value $(-1)^S \times 2^{E-b} \times (1 + 2^{-m} M)$; a subnormal ($E = 0$) has $(-1)^S \times 2^{1-b} \times (0 + 2^{-m} M)$.³ The hidden bit gives precision $p = m + 1$.¹⁰ Rounding to nearest bounds the relative error of any in-range normal value by $\varepsilon = 2^{-p}$ (Goldberg’s $(\beta/2)\,\beta^{-p}$ with $\beta = 2$).¹⁰

Range, precision and special values

The exponent field buys binades; the mantissa buys resolution within a binade. Two other choices change the count of usable values:

• Reserved codes. IEEE formats spend the all-ones exponent on $\pm\infty$ and NaN. That is a whole binade, which matters little at 32 bits and a lot at 8. OCP E4M3 keeps only S.1111.111 as NaN, which lifts its maximum from 240 to 448 and its range from 17 to 18 binades.² The MX FP6 and FP4 types reserve nothing at all.⁴
• Subnormals. They add $m$ binades of gradually decreasing precision below the smallest normal. In E4M3 they extend the bottom from $2^{-6}$ to $2^{-9}$.³ They cost hardware (a leading-zero count and normalization shift), which is why some datapaths flush them to zero.

Two consequences shape the rest of the chapter. First, a float’s error is relative: the absolute step at magnitude $|x|$ is about $2^{\lfloor \log_2 |x| \rfloor - m}$, so small values are represented as precisely (in relative terms) as large ones, until they fall out of the range. Second, the bias only positions the window; it doesn’t widen it. A tensor whose values span more binades than the format has will lose either its smallest or its largest values, whatever scale is applied. That is why range (exponent bits) and granularity of scaling are the two levers for low-precision formats.

Values from the BF16 study’s comparison table, the OCP FP8 specification and the OCP MX specification.⁷³⁴

Two splits of 15 bits

FP16 (E5M10, bias 15) and BF16 (E8M7, bias 127) spend the same 15 non-sign bits differently. FP16 has $p = 11$ and spans $2^{-24}$ to 65,504 (about 40 binades including subnormals); BF16 has $p = 8$ and spans the whole FP32 normal range, about $2^{-126}$ to $3.4 \times 10^{38}$.⁷ FP16 has 8× finer resolution within a binade; BF16 has 254 binades of normal numbers against FP16’s 30. Which one wins depends on whether a tensor’s problem is resolution or range.

• Gradients are range-limited. Micikevicius et al. found weight gradients whose exponents fell below −24 (about 5% of values in one network), and an SSD detector, many of whose activation gradients (recorded in FP32 training) fell below FP16’s smallest subnormal, that diverged in FP16 without scaling; a loss scale of 8 (three binades) restored FP32 accuracy.¹ Automatic mixed-precision tools adjust the loss scale at run time by provoking and detecting overflows.²
• Weight updates are resolution-limited. An update (learning rate × gradient) smaller than half an ulp of the weight vanishes under round-to-nearest, which is why the master weights stay in FP32 even with BF16.¹
• GEMM inputs tolerate either, as long as the products are summed at higher precision. Both the FP16 recipe and BF16 practice accumulate in FP32.¹⁷

BF16’s hardware argument is the multiplier: an 8-bit significand multiplier instead of 11 or 24 bits.⁷ Google states that a BF16 multiplier is about half the area of an FP16 one and one eighth of an FP32 one, because multiplier size scales with the square of the mantissa width.⁵ The $(p_{\mathrm{BF16}}/p_{\mathrm{FP16}})^2 = (8/11)^2 \approx 0.53$ and $(8/24)^2 \approx 0.11$ ratios agree with that to first order. The conversion from FP32 is a rounding of the low 16 bits; the BF16 study used round-to-nearest-even.⁷

OFP8 in detail

E4M3 has bias 7, no infinities and a single NaN mantissa pattern (S.1111.111), which extends it to 448 and 18 binades; E5M2 has bias 15 and IEEE-style specials (S.11111.00 is $\pm\infty$), reaching 57,344 and 32 binades.³ E5M2 is effectively FP16 with 8 fewer mantissa bits, which makes conversion trivial.² Conversion from wider types rounds to nearest even and then applies one of two overflow modes: saturating (clamp to $\pm\max$) or non-saturating (E4M3 → NaN, E5M2 → $\pm\infty$).³ Training recipes use saturation, because the loss-scaling trick of skipping a step on overflow would skip too many steps with FP8’s narrow range.²

The designers kept the IEEE-like biases and left scaling to software: a per-tensor scale in higher precision can take any real value, while a programmable exponent bias is equivalent to power-of-two scales only.² A scale chosen so that amax maps near the format’s maximum is applied before the cast, and its inverse is applied once per dot product (or folded into the next operation), so the cost is amortized over many multiply-accumulates.²

Integer quantization in detail

The affine scheme $r = S(q - Z)$ makes real zero exactly representable ($q = Z$) and expands an integer GEMM into an int8 × int8 core plus correction terms that cost $O(N^2)$ for an $N^3$ multiplication.⁹ Symmetric ($Z = 0$) quantization avoids the corrections, and an NVIDIA study found it sufficient for INT8 on all networks it examined.¹² The quantizer is

with two error sources: rounding error, uniform within $\pm s/2$ for in-range values, and clipping error for values outside $[s(q_{\min} - z),\ s(q_{\max} - z)]$. Raising $s$ trades the second for the first.¹³ Calibration picks $s$ from activation statistics (max, entropy, or a percentile such as 99.99%).¹² Granularity is the other knob: one scale per tensor, per output channel (common for weights), or per group of elements; finer groups generally improve accuracy at some cost in overhead.¹³

Float versus integer at 8 bits

For the same 8 bits, the difference between INT8 and FP8 is only where the grid points sit: uniform for INT8, denser near zero and sparser far out for FP8. So floats do better on distributions with heavy tails and outliers, and integers on uniform ones; on a Gaussian, the best low-exponent float formats and INT8 come out close.¹⁴ The simulator reproduces this: on its near-Gaussian “Weights” tensor (only two mild outliers), INT8 with a per-tensor scale has lower error than E4M3, and on the outlier-heavy “Activations” tensor the order flips.

Emergent outliers

Dettmers et al. found that in transformers, large-magnitude features (up to ~20× the other dimensions) appear first in about a quarter of layers and, around 6.7B parameters, in all of them. They occupy about 0.1% of feature dimensions; zeroing them degrades perplexity by 600–1000%.¹¹ They are highly systematic: at 6.7B parameters about 150,000 outliers per sequence fall in only 6 feature dimensions. LLM.int8() exploits that, sending those dimensions through a 16-bit GEMM and quantizing the rest to 8 bits with a separate scale per row and column.¹¹ Fine-grained block scaling attacks the same problem by limiting how many values each outlier can hurt.

Semantics

An MX format is a triple: scale type ($w$ bits), element type ($d$ bits) and block size $k$; a block costs $w + kd$ bits, and its memory layout is not prescribed.⁴ The concrete formats all use $k = 32$ and an E8M0 scale: unsigned, bias 127, exponents −127 to 127, a single NaN code (0xFF) that poisons the whole block, no infinities and no zero.⁴ E8M0 covers a superset of FP32’s exponent range.⁸ FP6 (E2M3, max 7.5; E3M2, max 28) and FP4 (E2M1, max 6) reserve no codes; conversion to them must support round-to-nearest-even and saturation to $\pm\max$. MXINT8 elements are two’s complement with an implicit $2^{-6}$ scale, i.e. a sign, one integer bit and six fraction bits.⁴

Conversion

emax_elem  = exponent of the element format's largest normal
             (E4M3: 8, E5M2: 15, E2M1: 2, E3M2: 4, E2M3: 2, INT8: 0)
shared_exp = floor(log2(max_i |V_i|)) - emax_elem
X          = 2^shared_exp                   # stored as E8M0: shared_exp + 127
P_i        = round_to_element(V_i / X)      # ties to even; clamp normals
                                            # above the element max to +/-max
value_i    = X * P_i

The rule follows the specification’s minimal semantics; implementations may use other algorithms.⁴⁸ The scale is shared along one axis, usually the reduction dimension of the matrix multiplication, so converting and transposing do not commute: a matrix that is reduced along different axes in the forward and backward passes has to be converted separately for each.⁸

The dot product

For two MX blocks, $\mathrm{Dot}(A, B) = X_A X_B \sum_i P_{A,i} P_{B,i}$: the element products are reduced first and the two power-of-two scales applied once. A longer dot product is a sum of block dot products and should produce an FP32 result; the internal precision and order of operations are implementation-defined.⁴ In hardware, the inner sum is a small fixed-point or narrow-float adder tree over 32 products, and the scale is an exponent addition before the result joins a wider accumulator.

The power-of-two penalty and NVFP4

Because $\lfloor \log_2 \mathrm{amax} \rfloor$ rounds down, $\mathrm{amax}/X$ lies in $[2^{e_{\max}}, 2^{e_{\max}+1})$. For E2M1 that is $[4, 8)$, and everything in $(6, 8)$ saturates to 6: the block maximum itself can lose up to 25%. Rounding the exponent up instead avoids clipping but leaves up to one binade of the tiny element range unused. NVIDIA’s NVFP4 paper describes MXFP4 as potentially losing up to one binade of dynamic range because of power-of-two scale rounding, and changes three things: 16-element blocks, E4M3 block scales with fractional precision, and a second, per-tensor FP32 scale that keeps the E4M3 scales in range. At least one value per block (its amax) is then represented at near-FP8 precision.¹⁷ The cost is $4 + 8/16 = 4.5$ bits per value instead of 4.25.

What 4 bits has been shown to do

• Inference: Dettmers and Zettlemoyer’s bit-level scaling laws (19M–176B parameters) found 4-bit weights almost universally optimal for accuracy per total model bit; below that, 3-bit reverses the trend.¹⁶
• Deployment: the gpt-oss models store their mixture-of-experts weights, over 90% of parameters, in MXFP4 at 4.25 bits per parameter.¹⁵
• Training: a 12B-parameter model pretrained on 10 trillion tokens with NVFP4 GEMMs reached 62.58% on MMLU-pro against 62.62% for an FP8 baseline. It needed Random Hadamard transforms to spread out block outliers, two-dimensional block scaling, stochastic rounding for gradients, and a few layers kept in higher precision.¹⁷

A first-order cost model

For a floating-point multiply-add the main blocks are: a $p \times p$ significand multiplier ($\text{area} \propto p^2$), an exponent adder ($\propto e$), an alignment shifter and adder for the accumulation ($\propto \text{accumulator width}$, with log-depth shifting), and normalization and rounding. For integers only the multiplier and a fixed-point adder remain. The survey by Sze et al., citing Horowitz’s ISSCC 2014 figures, states the scaling explicitly: multiplier energy and area scale roughly quadratically with bits, adder and memory cost roughly linearly.⁶

This is the multiplier array only. It explains why BF16 multipliers are about half of FP16’s and an eighth of FP32’s,⁵ but it overstates the savings for tiny floats, whose cost is dominated by the shifters, the exponent logic and above all the accumulation. The van Baalen et al. gate-count study makes the point: FP8-E4 with a fixed-point accumulator is 53% more expensive than INT8, and with an FP32 accumulator 183% more. Their INT8 design needs a 27-bit fixed-point accumulator (15 bits of product plus 12 bits of headroom for 4,096 products); FP8-E4, whose products span many more binades, would need 37 bits.¹⁴ That is consistent with published peak figures, where halving the input width roughly doubles matrix throughput rather than quadrupling it.¹²¹⁹

Accumulation error

Floating-point accumulation is inexact and order-dependent, and the error grows with the reduction length $K$.¹⁴ The failure mode is swamping: when the running sum exceeds an addend by more than $2^{m+1}$, the addend is shifted out entirely, and the problem worsens with long sums and non-zero-mean data.¹⁸ Wang et al. showed that splitting a dot product into chunks summed separately and then combined (chunk-based accumulation), plus stochastic rounding, let FP8 training use 16-bit additions.¹⁸

A real-world example: DeepSeek-V3’s authors measured that FP8 GEMMs on the GPU they used retain only about 14 bits of accumulation precision inside the matrix unit, giving a maximum relative error of nearly 2% for $K = 4096$. Their fix copies partial sums to FP32 registers every 128 elements of $K$, which is chunked accumulation implemented in software.²⁰ The general lesson is that the narrow-format story is really two stories: input precision, which sets multiplier cost and bandwidth, and accumulation precision, which sets numerical quality and is much cheaper to keep high.

Encodings follow OFP8 and MX v1.0 exactly: round-to-nearest-even, subnormals, saturation, E8M0 scales computed as $2^{\lfloor \log_2 \mathrm{amax} \rfloor - e_{\max,\mathrm{elem}}}$. FP8 and integer formats use an FP32 per-tensor scale mapping amax to the format maximum. Click bits to decode arbitrary patterns (try S.1111.111 in E4M3, or the all-ones exponent in E5M2). The block-size slider sweeps MX from $k = 8$ to 256: MXFP4 on the activations goes from 18% to 32% to 52% RMS error of $\sigma$ as $k$ grows from 8 to 32 to 256, while the scale overhead falls from 1 to 0.03 bits per value. Watch the clipped count too: a block maximum just below the next power of two saturates in E2M1 and E4M3 alike. The readouts include SQNR and a $p^2$ multiplier-cost estimate.

E4M3 / E5M2 dynamic range

18 / 32 binades

FP8 GEMM accumulation measured on one GPU

≈ 14 bits

NVFP4 vs FP8 pretraining, MMLU-pro (12B, 10T tokens)

62.58% vs 62.62%

MXFP4 share of gpt-oss parameters

over 90%

Format parameters

From the FP16 column of the BF16 study, the OFP8 specification and the MX specification.⁷³⁴

Published results worth knowing

• FP8 (E4M3 forward, E5M2 gradients, per-tensor scales) matched 16-bit training on CNNs, RNNs and transformers up to 175B parameters with unchanged hyperparameters.²
• DeepSeek-V3 (671B total parameters) trained with FP8 GEMMs, E4M3 on all tensors, 1 × 128 activation tiles and 128 × 128 weight blocks; in tests at two smaller scales the FP8 model’s loss stayed within 0.25% of a BF16 baseline.²⁰
• 8-bit MX direct-cast inference on FP32-trained models with minimal loss; 6-bit MX training at FP32 quality.⁸
• Emergent outliers in LLMs: about 0.1% of features, up to 20× larger, in all layers from about 6.7B parameters.¹¹

The design space

• Element width and type. At fixed bits, exponent versus mantissa trades range for SQNR (about 6 dB per mantissa bit). Integers maximize resolution for a known range; floats tolerate unknown or heavy-tailed distributions.¹⁴²²
• Scale granularity and type. Per-tensor FP32 scales are free in storage but expose every value to the tensor’s worst outlier. Per-block scales cost $8/k$ bits per value; power-of-two scales cost only an exponent add but can waste up to a binade; E4M3 scales are more precise but need a second-level tensor scale for range.⁴¹⁷
• Accumulator width. Exact fixed-point accumulation is cheapest for integers; float inputs with many exponent bits need very wide fixed-point accumulators or a floating accumulator, which is where FP8 loses much of its hardware advantage over INT8.¹⁴
• Software burden. Every step down the ladder has added machinery: master weights and loss scaling for FP16, per-tensor scaling for FP8, tile and block scaling plus promoted accumulation in FP8 training at scale, and Hadamard rotations, stochastic rounding and high-precision layers for FP4 training.¹²⁰¹⁷

Failure modes in detail

• Range failures are binary and data-dependent: a tensor that fits during calibration may overflow on an unusual input. Saturating conversion bounds the damage; non-saturating conversion turns it into NaN or $\infty$.³
• Resolution failures are statistical: SQNR drops gradually, and the effect on accuracy depends on the layer. Operations other than matrix multiplications, such as nonlinearities and normalizations, typically run in higher precision anyway,² and the NVFP4 recipe also kept a few linear layers in higher precision for stability.¹⁷
• Systematic bias: round-to-nearest of updates smaller than half an ulp is always zero, so many small updates are lost rather than averaged; stochastic rounding removes that bias at the cost of noise and a random-number source.¹⁸
• Hardware-specific numerics: the MX spec leaves dot-product internal precision implementation-defined, so the same model in the same format can produce slightly different results on different chips.⁴ DeepSeek-V3’s accumulation measurement is an example of such a detail mattering at scale.²⁰

Rounding to a small float

Every format in this chapter can be produced by one routine: find the binade, round the significand to $m$ bits with ties to even, saturate, and handle the bottom of the range with subnormals. This is what the simulator runs, per the OFP8 and MX conversion rules.³⁴

emin = 1 - bias                      # exponent of the smallest normal
a    = |x|
e    = max(floor(log2(a)), emin)      # below emin: subnormal spacing
ulp  = 2^(e - m)                      # step size in this binade
r    = round_half_even(a / ulp) * ulp
if r > max_normal: r = max_normal     # SAT mode; NONSAT gives Inf or NaN
return sign(x) * r                    # r == 0 means the value underflowed

Noise model for a uniform grid

If values fall at random positions between grid points spaced $\Delta$ apart, the rounding error is uniform on $[-\Delta/2, \Delta/2]$ and its mean square is $\Delta^2/12$. For a symmetric $b$-bit integer quantizer with a per-tensor scale set by the tensor’s largest magnitude, $\Delta = \mathrm{amax} / q_{\max}$ with $q_{\max} = 2^{b-1} - 1$. For values with RMS $\sigma$, the signal-to-quantization-noise ratio (SQNR) is

Three readings of this formula:

• Each extra bit doubles $q_{\max}$ and adds $20 \log_{10} 2 \approx 6.02$ dB.
• Each doubling of the crest factor $\mathrm{amax}/\sigma$ costs the same 6 dB, a full bit. An outlier 20× the RMS costs about 26 dB, more than four bits.
• For INT8 with $\mathrm{amax}/\sigma \approx 21$, as in the simulator’s activation tensor, the formula predicts $52.9 - 26.4 \approx 26.5$ dB on the ordinary values; the simulator measures an RMS error of 4.8% of $\sigma$, which is 26.3 dB.

Widening the range past $\mathrm{amax}$ is never useful, but narrowing it below $\mathrm{amax}$ trades this rounding noise for clipping error on the tail: the total error is roughly $\Delta(\alpha)^2/12$ plus the mean of $(|x| - \alpha)^2$ over values beyond the clipping point $\alpha$. Calibration methods choose $\alpha$ to balance the two.¹³¹²

Noise model for a float

Within a binade $[2^e, 2^{e+1})$ the step is $2^{e-m}$, so the error is still uniform, with mean square $2^{2(e-m)}/12$. Relative to $x^2$, and averaging $1/x^2$ over a value spread evenly through the binade (the mean of $1/u^2$ for $u$ uniform on $[1, 2)$ is $1/2$), the relative noise power is $2^{-2m}/24$. That gives, independent of scale while values stay in range,

The linear dependence on mantissa bits matches the analysis of block formats by Rouhani et al.²² The prediction is 31.9 dB for E4M3 ($m = 3$), 25.8 dB for E5M2 and 55.9 dB for BF16; the simulator measures 32–34, 26–28 and 54–62 dB on its three tensors. The model breaks down at the edges: values pushed into the subnormal range lose relative precision, values below the smallest subnormal are lost, and values above the maximum clip. That is why a float’s SQNR is flat across scales only within its dynamic range, and why the scale factor for FP8 matters even though the float grid is “self-scaling.”

Block scaling

Block scaling replaces the tensor’s crest factor with each block’s. For an MX block with scale $X = 2^{\lfloor \log_2 \mathrm{amax} \rfloor - e_{\max}}$, the block maximum lands in $[2^{e_{\max}}, 2^{e_{\max}+1})$ of the element format: on average the top of the element range is under-used by about half a binade, and in FP4 a maximum in $(6, 8) \cdot X$ saturates. Shared-exponent analyses show the resulting SQNR growing linearly with mantissa bits and logarithmically with finer block granularity.²² The cost side is exact: $8/k$ bits per value for an E8M0 scale. In the simulator, MXFP4 on the activation tensor falls from 18% to 32% to 52% RMS error of $\sigma$ as $k$ goes from 8 to 32 to 256, for 1, 0.25 and 0.03 bits of overhead.

Accumulation error

Summing $N$ terms one after another in a float with unit roundoff $u = 2^{-p}$ gives an error bound that grows in proportion to $N$. Splitting the sum into chunks of length $\mathit{CL}$, summing each, and then summing the partial sums reduces the bound to order $N/\mathit{CL} + \mathit{CL}$,¹⁸ which is smallest when $\mathit{CL} \approx \sqrt{N}$. For $N = 4096$ that is $\mathit{CL} = 64$; DeepSeek-V3’s promotion interval of 128 sits in the same range, and it was their answer to errors of nearly 2% measured at $K = 4096$.²⁰

For integers the question is width, not rounding: a $b$-bit × $b$-bit product needs $2b - 1$ bits, and summing $2^g$ of them needs $g$ more. The INT8 example in van Baalen et al. is $15 + 12 = 27$ bits for 4,096 products; a fixed-point accumulator that exactly covers FP8-E4 products would need 37 bits.¹⁴ That is why 32-bit integer accumulators are standard for INT8.⁹

Stochastic rounding

Round-to-nearest maps every value in $(x_k - \mathrm{ulp}/2,\ x_k + \mathrm{ulp}/2)$ to $x_k$, so an update smaller than half an ulp of the weight is lost every time, however often it is applied. Stochastic rounding instead rounds up with probability $(x - \lfloor x \rfloor)/\mathrm{ulp}$, so $E[\mathrm{round}(x)] = x$ and small updates survive on average. Wang et al. used it to make 16-bit weight updates work in FP8 training,¹⁸ and the NVFP4 recipe uses it for gradients.¹⁷