Comparing chips

You know a neural network is mostly matrix multiplication, and the earlier chapters covered how GPUs, systolic arrays, spatial meshes and wafer-scale designs execute it. This chapter is about the measurement problem: turning “chip X has N TFLOPS” into a defensible statement about delivered work per dollar.

The pieces, each made precise below:

• Peak versus delivered. Utilization and its training-specific form, MFU, with published values from 21% to 58%, and why inference often sits lower.²
• Benchmark methodology. MLPerf’s divisions, availability categories, inference scenarios, time-to-train scoring and the statistical rules that stop cherry-picking.⁶⁷
• Power boundaries. Chip TDP, measured chip power, node wall power, rack and facility (PUE), and why “incremental” and “total” perf/W can differ by more than 2×.¹
• TCO models. Capex amortization, energy opex, duty cycle and idle power, reduced to dollars per million tokens.
• Claim hygiene. Sparsity-doubled peaks, precision mismatches, batch-size and latency choices, system-size choices, and what MLCommons’ own messaging rules forbid.⁹

Peak is $\text{units} \times \text{FLOPs/clock/unit} \times f_{\mathrm{clk}}$ at a stated precision and clock. Two vendor whitepapers show the arithmetic: A100 = 108 SMs × 1,024 FP16 FMA/clock × 2 × 1.41 GHz = 311.9 TFLOP/s;¹⁴ MI300X = 304 CUs × 2,048 FP16 FLOPs/clock × 2.1 GHz = 1,307.4 TFLOP/s.¹⁵ Note what this assumes: every MAC issues every cycle at the boost clock, with no stall, no non-matrix instruction and no power or thermal throttling.

Utilization: be precise about the ratio

“Utilization” is ambiguous until you name the numerator. PaLM’s authors separated two:²

• Hardware FLOPs utilization (HFU): FLOPs the hardware actually executed ÷ peak. It counts rematerialization (recomputing activations to save memory), so a run can raise HFU by doing more work without training any faster. It also depends on how FLOPs are counted (analytically or with performance counters).
• Model FLOPs utilization (MFU): observed tokens/s × the FLOPs the model mathematically needs per token ÷ peak. For a dense decoder that is about 6N per training token (2N forward, 4N backward) plus attention. Its numerator is a pure throughput measurement.

PaLM 540B ran at 46.2% MFU and 57.8% HFU on 6,144 TPU v4 chips. Recomputed on the same basis, GPT-3 was 21.3%, Gopher 32.5% and Megatron-Turing NLG 530B 30.2%.² Megatron-LM reported 52% of theoretical peak per GPU at 3,072 GPUs, counted with its own analytical FLOP formula.³ Treat cross-paper comparisons with care: the peak in the denominator, the attention FLOPs and the recomputation accounting all vary.

Why inference often sits lower

Hardware counters on the first TPU make the causes concrete. Across six production workloads it averaged 21.4 of 92 TOPS. Its two convolutional networks reached 86.0 and 14.1 TOPS; its two LSTMs reached 3.7 and 2.8; for the four memory-bound MLP and LSTM workloads, 44–62% of cycles were weight-load stalls.¹ Latency adds a second ceiling: under a 7 ms 99th-percentile bound for one MLP, a CPU ran at 42% and a GPU at 37% of the throughput they reached with relaxed latency, because the bound forced batch 16 instead of 64.¹ The same two forces (low arithmetic intensity in decode, and tail-latency limits on batching) govern LLM serving today; see The memory wall.

The rules are public on GitHub and are the real specification; the summaries below follow the versions checked in October 2026.

Inference

• Architecture. A fixed load generator (LoadGen) issues queries to the submitter’s system under test and timestamps the answers; data set and accuracy script are also fixed. Everything behind the interface (framework, kernels, batching, scheduling) is the submitter’s choice.⁴
• Scenarios and metrics. Single stream: 90th-percentile latency. Multistream: 99th-percentile latency of 8-sample queries. Server: Poisson arrivals; the metric is the highest arrival rate at which the benchmark’s tail-latency bound holds. Offline: one query holding at least 24,576 samples; the metric is throughput. Minimum duration 600 s.⁶
• LLM latency bounds. For Llama2-70b, the Server bound is a time to first token (TTFT) of 2,000 ms and a time per output token (TPOT) of 200 ms; a separate Interactive category tightens them to 450 ms and 40 ms. Its accuracy target is 99.9% of the FP32 reference’s ROUGE scores, with output length at least 90% of the reference.⁶
• Closed-division quantization. Any “purely mathematical, reproducible” quantization to any format is allowed if it meets the quality target, uses only the supplied calibration set and is publicly described. No retraining. Datacenter systems must run with ECC enabled on DRAM and HBM.⁶
• Divisions. Closed, Open, and a Network division for systems whose queries arrive over a network.⁶

Training

• Metric. Time to train to a quality target, timed from first touching training data. Larger batches can raise throughput yet need more epochs: in the v0.5 round ResNet-50 took about 64 epochs at a 4K minibatch but over 80 at 16K, about 30% more computation.⁵
• Scoring. Each result is a set of $N$ runs (10 for most current benchmarks, 3 for the largest LLMs); the fastest and slowest are dropped and the rest averaged. Running many sets to pick the best is prohibited, and runs that converge faster than the reference convergence points allow are rejected or rescaled.⁷
• Precision. Closed training allows a pre-approved list of formats (FP32 down to FP8, MXFP4/MXFP6, NVFP4, INT8/INT4 and others) and only “numerically safe” compensations such as scaling. From v6.1, every log must record the lowest precision used in linear layers, attention and communication.⁷

Availability and review

Available hardware must have pricing (public or on request) and have shipped to at least one third party. A Preview system must be resubmitted as Available in the round after 140 days (or the next round, whichever is later) with performance within 2%, or the Preview result is marked invalid. RDI components can’t reappear as Available until the round after next or 221 days, whichever is longer. Review is by a committee that includes a representative of every submitter, which “operates on balance of interests rather than by avoiding conflict of interest”: competitors audit each other.⁸

What MLPerf does not fix: system size. Vendors choose how many chips to submit, and as the TPU v4 authors noted, ideally systems would be compared at equal size, cost or power, “but that is not required.”¹⁶ Price is not part of any result.

Define the boundary before the ratio. From the inside out:

Rated versus measured. The TPU v4 paper lists TDP as “N.A.” and instead reports measured minimum/mean/maximum chip-plus-HBM power on production applications (121/170/192 W).¹⁶ LBNL’s 2024 report cites measurements of 8-GPU servers rated at 10.2 kW that drew about 7.9 kW (≈78% of rated) in computationally saturated training, idled near 18% of rated, and models annual operation at 70% of rated.¹⁷

MLPerf Power’s choices. Datacenter submissions measure full-system power over the timed execution phase with a SPEC-certified analyzer or, at scale, hardware counters plus estimates for switches. Cooling and storage nodes are not yet included. The power group explicitly rejects three shortcuts: isolating accelerator power (“Myth #1”), using TDP or PSU ratings (“Myth #2”), and folding in PUE (“Myth #3”), because PUE describes the building, not the ML system, and can’t be verified across submitters.¹¹

Total versus incremental. For an accelerator added to a host, define $\text{total perf/W} = \text{perf} / (P_{\mathrm{host}} + P_{\mathrm{acc}})$ and $\text{incremental perf/W} = \text{perf} / P_{\mathrm{acc}}$. Jouppi et al. report both: the TPU server at 17–34× a CPU server on total, 41–83× on incremental; the GPU server at 1.2–2.1× and 1.7–2.9×.¹ Incremental is the right metric only if the host would exist anyway; in a purpose-built AI server it flatters the accelerator.

Accuracy target interacts with energy. On BERT in MLPerf Inference v3.1 and v4.0 datacenter Offline results, moving from the 99% to the 99.9% accuracy target cut samples per joule by about half on average, though newer quantization methods narrowed the gap.¹¹

A per-accelerator hourly model, which is what the simulation below implements:

Here $L$ is the depreciation life in years, $b$ the busy fraction (duty cycle), $k_{\mathrm{idle}}$ the idle draw as a fraction of loaded draw, and $U$ utilization. $\mathrm{FLOP}_{\text{per token}} \approx 2N$ for decoding a dense $N$-parameter model. LBNL cites a measured 8-GPU node idling at about 18% of rated power and puts loaded draw at roughly 0.7–0.8 of rated, which makes $k_{\mathrm{idle}}$ about 0.25; the simulation uses 0.2.¹⁷

Three structural consequences:

1. Utilization and duty cycle divide capex directly. $\$/\text{token} \propto 1/(U b)$ for the capex term. Halving utilization through a software regression costs as much as doubling the price.
2. Energy share $= C_{\mathrm{en}} / (C_{\mathrm{cap}} + C_{\mathrm{en}})$. With $25–40k capex, ~1.2–1.6 kW per accelerator, PUE 1.3 and $0.10/kWh it is roughly 12–23% (15% for chip A and 18% for chip B in the “Faster on paper” preset). It approaches half only when capex per watt-year is small: cheap chips, long lives, or electricity well above $0.30/kWh.
3. The ranking can flip with site parameters. A chip that is faster but power-hungry wins where power is cheap and loses where it is dear; try the “Power bill” preset and then lower the electricity price.

What the simple model omits matters at scale: network fabric and switches per accelerator (see Scale-out networking), facility capex per kW of critical power, failure and repair, software engineering to reach the assumed U, and financing. Power-limited sites change the objective entirely: when megawatts rather than dollars are the binding constraint, work per facility watt (the system-level perf/W) becomes the figure of merit.

A checklist, with what each trick does to the number:

Two concrete headline numbers show the pattern. The DGX A100 table lists “5 (GPU Tensor PFLOP)” for 8 GPUs: 8 × 624 TFLOPS, the sparse FP16 rate; the dense figure is 2.5 PFLOPS.¹⁴ The MI300X table’s largest entry is 5,229.8 TFLOPS: FP8 with sparsity, 4× its dense BF16 rate.¹⁵ Both are accurately labeled in the whitepapers; the distortion happens when one vendor’s sparse FP8 figure is set beside another’s dense BF16.

On means: the first TPU paper reported the TPU die 14.5× a contemporary CPU die by geometric mean over six applications, but 29.2× when weighted by the actual production mix; for the GPU the two means were 1.1× and 1.9×.¹ Neither is wrong; they answer different questions. The same paper rebutted, in its fallacies section, the claim that its competitors would have looked better if run differently, which is a reminder that vendor studies, this one included, are written by interested parties.

MLCommons’ messaging guidelines make several of these rules binding for MLPerf results: comparisons must identify any difference in version, division, category, verified status, scenario or chip count; MLPerf results may not be compared with non-MLPerf results; derived metrics (cost, perf/W from anything but measured MLPerf power) must be marked as such.⁹

The model is the hourly TCO equation from the previous section, with $\mathrm{FLOP}_{\text{per token}} = 2 \times 70 \times 10^9$ and idle draw at 20% of loaded. Expert view adds the duty cycle, energy per token, lifetime capex and energy, and the worked equation for the chip being edited. Try:

• The “Idle fleet” preset (30% busy). How does the capex share change, and does the ranking?
• Find the break-even electricity price in “Power bill”, and compare it with the EIA averages below.
• Make A and B equal except utilization and price. What price ratio offsets a 1.5× utilization advantage?

Everything is illustrative: no real chip’s price or utilization is implied, and the model omits network, facility capex, staff and software.

• Goodhart’s law on benchmarks. MLPerf’s defenses (reference convergence points, run minimums with fastest and slowest dropped, compliance tests, and peer review by competitors) limit overfitting, but tuning effort concentrates on scored models.⁷⁸ The same-chip spreads in the v6.1 table are a direct measure of how much the software and system stack moves the result.¹³
• Scale choice. Single-node results hide interconnect behavior; rack-scale results fold in the fabric, which is often where the real differences lie (see Scale-up fabrics). Compare at the scale you will deploy.
• Accuracy target as a knob. Lower targets permit more aggressive quantization; MLPerf Power’s BERT data show the 99.9% target roughly halving average samples per joule versus 99%.¹¹ Make sure two claims use the same quality bar.
• Power capacity versus capex. When the site’s power budget is the binding constraint, optimize delivered work per facility watt; when capital is binding, optimize $/token. These can select different chips from the same data.
• Failure modes in TCO models. Assuming peak instead of delivered throughput, 100% duty cycle, TDP instead of measured power, and list price instead of system price each bias the answer in a predictable direction; a model that assumes all four can be off by several times.

1. MFU from first principles

For a dense decoder-only transformer with $N$ parameters, $L$ layers, $H$ heads, head dimension $Q$ and sequence length $T$, PaLM’s Appendix B gives training FLOPs per token as $6N + 12LHQT$.² With a system peak of $P$ FLOP/s, the theoretical maximum is

and

Worked example from the paper: Megatron-Turing NLG 530B trained at 65,430 tokens/s on 2,240 A100s with 312 peak TFLOP/s each. Ignoring attention:

or 30.2% with attention.² For inference decode, replace $6N$ with about $2N$ per generated token; the same structure applies, but decode is usually bandwidth-bound, so a low MFU there is expected rather than a failure.

2. Early stopping in the Server scenario

A latency percentile estimated from finitely many queries is uncertain. MLPerf Inference treats each query as a Bernoulli trial that meets the latency bound with unknown probability $p$. After $t$ over-latency queries, it finds the smallest number of under-latency queries $h$ such that a “minimally failing” system (one that meets the bound with probability exactly the target percentile minus a tolerance $d$) would produce a result this good with probability at most $1 - c$:

equivalently

where $F$ is the binomial cumulative distribution and $I$ the regularized incomplete beta function, solved by binary search. MLPerf uses $d = 0$ and $c = 0.99$. A system comfortably inside its bound stops early; one close to it must run more queries; one outside it never passes.⁶ For a 90th-percentile target with 99% confidence and 0.5% margin, the suggested minimum is 23,886 inferences, rounded to 24,576 ($3 \times 2^{13}$).⁶

3. Olympic scoring and convergence checks in training

Time to train is a random variable: initialization and data order change how many epochs a run needs. MLPerf Training therefore runs $N$ independent seeds ($N = 10$ for most current benchmarks), discards the fastest and slowest, and averages the rest; one non-converging run may be dropped as the slowest, two invalidate the result. Submitters may run $M > N$ runs only if they pick the $N$ consecutive runs (by an objective order such as timestamp) whose score is closest to the median, which prevents choosing the luckiest set. Reference convergence points, measured from the reference implementation at several batch sizes, set a floor on epochs to converge; submissions that converge statistically faster are rejected or have their scores scaled.⁷

4. Why Server throughput falls below Offline

Offline gives the system every sample at once, so it can run the largest batch that fits in memory and keep the pipeline full. Server sends Poisson arrivals at rate $\lambda$; to keep the 99th-percentile time to first token under its bound, the scheduler must start new requests promptly and cap the batch so that time per output token stays under its bound too. Queueing raises tail latency sharply as utilization approaches 1, so the reported $\lambda$ sits where the tail, not the mean, hits the limit; the first TPU paper makes the same point about input queues raising throughput but stretching response time.¹ In the v6.1 Llama2-70b data, the two 8-accelerator vendor systems in the table kept 97–98% of Offline throughput (other 8-accelerator servers in the round kept roughly 92–100%), while a 4-card system kept 71%.¹³

5. A complete TCO comparison, step by step

Using the simulation’s “Faster on paper” preset (all values hypothetical):

A has twice B’s peak and 1.4× B’s peak perf/W per chip watt (2.0 vs 1.43 TFLOP/s/W), yet B delivers 14% more tokens per chip, 51% more per facility watt, and costs 43% less per token. Break-even for A on cost requires $U \approx 39\%$, or a price near $19,500 at 22% utilization. The arithmetic is trivial; the hard part in practice is measuring $U$ and power honestly on your workload.

Most of the terms this chapter left as inputs are outputs of system design: server overhead watts, the switch power MLPerf Power finds hard to attribute, the scaling efficiency behind rack-scale results, and the PUE of a liquid-cooled hall. The Systems guide covers them from the package up: Packaging and chiplets, Board and server, Scale-up fabrics, Scale-out networking, Optics, parallelism and Power and cooling.