The memory wall

The roofline says attainable throughput is the lesser of peak compute and bandwidth × arithmetic intensity.³ The memory wall is the observation that the first term has grown much faster than the second for decades, so the ridge point keeps moving right and more real kernels fall on the bandwidth slope.¹ The wall has three faces: bandwidth (bytes per second to the compute), capacity (bytes that fit close enough to be fast), and latency, which has improved least of all.

LLM inference shows all three. The decode phase does about 2 FLOPs per parameter per token while reading every parameter once per step, an intensity of 1–2 FLOPs per byte at batch 1 against ridge points of several hundred on current accelerators. Batching raises the weight-read intensity linearly, but each sequence also carries a KV cache that is read every step, is not shared across the batch, and competes with the weights for capacity.¹⁷

This chapter quantifies the trend, then walks the hierarchy from HBM to on-chip SRAM, builds the decode model (memory time versus compute time, the batch “knee”, the capacity bound), and evaluates the mitigations: batching and its scheduling, quantization of weights and KV cache, grouped-query attention, speculative decoding and splitting the phases. The full model, with its assumptions, is in “Under the hood”.

Rates, compounded

Fitted over about twenty years of server-class AI hardware, peak FLOPS grew 3.0× per two years, DRAM bandwidth 1.6× and interconnect bandwidth 1.4×, which over the period is 60,000×, 100× and 30×.¹ The ratio of the first two, about 1.9× per two years, is the rate at which the ridge point drifts right. Two consequences:

• A fixed algorithm slides down the roofline over generations. A kernel at 100 FLOP/B that was compute-bound on a chip with a 60 FLOP/B ridge becomes memory-bound when the ridge passes 100, without any change to the code.
• Scaling out doesn’t escape it. Spreading a model over more chips trades on-chip traffic for chip-to-chip traffic, which has grown slowest of the three.¹ See Why the network looks this way in Systems.

Capacity scales no better. Between 2018 and 2022 the parameter count of state-of-the-art Transformers grew about 410× per two years against about 2× for single-accelerator memory, so every frontier model is now partitioned.¹ Latency, the third face, has been even harder to improve than bandwidth.¹ Accelerators therefore hide it behind many outstanding requests (many warps on a GPU, long DMA streams on a systolic design) rather than wait on it.

Energy per access by level

Sze, Chen, Yang and Emer normalize the energy of fetching an operand to the energy of the ALU operation itself (1×): about 1× from the register file, 2× from a neighboring PE over the array network, 6× from a 100–500 kB global buffer and 200× from DRAM.² Two orders of magnitude between a small on-chip SRAM and DRAM is the reason dataflow design exists: a weight fetched once from DRAM and reused 200 times from local storage costs about the same energy per use as the arithmetic. The same logic gives one roofline per level. A kernel tiled for on-chip SRAM is judged against the SRAM bandwidth ridge; the DRAM ridge only binds for the traffic that actually leaves the chip.

Intensity of the transformer phases

Counting a multiply-accumulate as 2 FLOPs, a dense layer applied to $n$ tokens does $2n$ FLOPs per weight. If the weight is read once in $b$ bytes, intensity from weights alone is $2n/b$.

• Prefill processes every prompt token together, so $n$ is the prompt length times the batch: thousands of FLOPs per byte, above any current ridge.
• Decode processes one token per sequence per step, so $n = B$, the batch size. At $B = 1$ with 16-bit weights the intensity is 1 FLOP/B.

Profiling makes the same point: GPT-2 and BERT-Base have nearly the same configuration and total FLOPs, yet the autoregressive model is much slower, because its matrix–vector steps have far lower intensity.¹ The chips in “By the numbers” below have ridge points from about 140 to about 330 FLOP/B.

Bandwidth arithmetic

$\text{Per-stack bandwidth} = \text{interface width} \times \text{per-pin rate} / 8$. Over the generations, both terms moved:

• First-generation HBM, as shipped in 2015, ran 1 Gb/s per pin, so four stacks on a 4,096-bit interface gave 512 GB/s, 128 GB/s per stack.⁵ HBM2 reached 2.4 Gb/s for 307 GB/s; HBM3 runs 6.4 Gb/s for 819 GB/s; HBM3E parts run 9.8 Gb/s for more than 1.2 TB/s.⁶⁷⁸
• JEDEC’s JESD270-4 (HBM4, April 2025) doubles the width to 2,048 bits and the independent channels from 16 to 32 (each split into 2 pseudo-channels), at up to 8 Gb/s, for up to 2 TB/s per stack. It defines 4-, 8-, 12- and 16-high stacks of 24 Gb or 32 Gb dies, up to 64 GB.⁹

The host side mirrors this. H100 SXM5 drives five HBM3 stacks from ten 512-bit memory controllers, a 5,120-bit interface, for over 3 TB/s at launch; that works out to roughly 4.7 Gb/s per pin, below the HBM3 maximum.¹⁰⁷ MI300X uses eight HBM3 stacks for 5.3 TB/s.¹¹

Constraints that come with it

• Shoreline. Each stack’s PHY occupies die edge on the processor, and the stack must sit within a few millimeters of it. Stack count is therefore bounded by the processor’s perimeter.
• Interposer area. Logic die plus stacks must fit on the interposer, whose size, yield and cost are packaging problems in their own right (see Packaging and chiplets).
• Capacity and bandwidth are coupled. More stacks give both; taller stacks give capacity only. Doubling capacity at constant bandwidth halves how often the chip can read all of its memory, which matters for decode, where every step reads most of it.
• Peak is not sustained. Refresh, bank conflicts and read/write turnarounds take a share; streaming kernels typically reach a large fraction of peak but not all of it, so the decode model in the simulation is an upper bound.

The scaling stall

At the 3 nm generation the high-density SRAM bitcell stopped shrinking. TSMC’s N3 cell is 0.0199 µm², and the N3E variant, which is believed to relax some pitches for performance and yield, discloses 0.021 µm², larger than N3’s.¹⁴ Trade press summarized the 2022 disclosures as SRAM scaling stalling at 3 nm, with direct effects on the die size and cost of high-performance chips such as GPUs and processors; scaling resumed to some degree at 2 nm, TSMC’s and Intel’s first nanosheet node.¹⁵¹⁶ The bitcell is built from the smallest devices in the process, and its read and write margins depend on how well they match (the butterfly curve and static noise margin in Memory cells), one reason it is harder to shrink than logic.

Area budget arithmetic

Take the densest SRAM blocks TSMC and Intel reported at ISSCC 2025, 38.1 Mb/mm² on 2 nm-class processes.¹⁶ Real arrays lose density to sense amplifiers, decoders, redundancy and the wiring that connects banks to compute, so treat the following as a lower bound on area:

• $1\,\mathrm{GB} = 8 \times 10^3\,\mathrm{Mb}$ → about 210 mm².
• A 256 MiB last-level memory (MI300X’s size) → about 56 mm², already a meaningful slice of an advanced-node die budget.
• A 70B model at 8 bits → about 15,000 mm²: dozens of reticle-class dies.

Hence the two strategies. Most accelerators keep weights in HBM and size SRAM for reuse: enough to hold tiles of weights, activations and KV blocks so that each HBM byte is used many times. SRAM-heavy and wafer-scale designs instead keep weights entirely on chip and pay for capacity with chip count, which turns the memory wall into an interconnect problem (see Wafer-scale and SRAM-heavy designs).

Two rooflines

With an SRAM level, each kernel has two intensities: FLOPs per HBM byte and FLOPs per SRAM byte, each judged against its own ridge. FlashAttention’s contribution is to make attention IO-aware: it tiles the computation so that the $N \times N$ score matrix lives only in SRAM, cutting HBM reads and writes.⁴ That moves attention right on the HBM roofline. It does not change what decode must read from HBM each step, which is set by the weights and the KV cache.

Memory time and compute time

Pope et al. give the standard first-order model. Weights and KV cache are the two large tensors, and each is transferred from HBM to the compute once per forward pass; that transfer time is the “memory time”. An $N$-parameter decoder needs $2N$ matmul FLOPs per token; at peak that is the “compute time”.¹⁷ For one decode step at batch $B$ and context $L$:

• Bytes: $W + BLk$, with $W = N \times \text{bytes per weight}$ and $k$ the KV bytes per token.
• FLOPs: $B\,(2N + 4\, n_{\mathrm{layers}} L\, d_{\mathrm{model}})$; the second term is attention ($QK^{\mathsf{T}}$ and the weighted sum over $V$), small next to $2N$ until $L$ reaches thousands.
• Step time $\approx \max(\text{bytes} / \mathrm{BW},\ \text{FLOPs} / \text{peak})$; per-user rate $1 / \text{step time}$; aggregate $B / \text{step time}$.

At small $B$ and $L$ the $W$ term dominates; at large $B$ and $L$ (Pope et al. cite 2,048+ tokens at batch 512+) the KV term does.¹⁷ For a 500B+ model with full multi-head attention at batch 512 and context 2,048, the KV cache totals 3 TB, three times the parameters, and must be read once per generated token while the compute is essentially idle.¹⁷

Why batching stops helping

The weight term’s intensity is $2B / b_{\mathrm{w}}$, so it rises with batch. The attention term’s is not: per layer and sequence it does $4Ld$ FLOPs and reads $2L\, g\, d_{\mathrm{h}} b_{\mathrm{kv}}$ bytes ($g$ KV heads of size $d_{\mathrm{h}}$). With $h$ query heads, $d = h\, d_{\mathrm{h}}$, so

independent of both $B$ and $L$. For Llama 3 70B ($h/g = 64/8$) at 16 bits that is 8 FLOP/B.²⁰ As $B$ grows, the step’s blended intensity levels off at one sequence’s FLOPs per KV byte, and as $L$ grows that ratio falls toward this small number, so at long context decode stays memory-bound at any batch size. The simulation below shows this as a curve that never reaches its compute roof.

Capacity sets the batch, and fragmentation wastes it

The largest batch that fits is $B_{\max} = (C - W) / (Lk)$. Static per-request allocation wastes much of that: in the systems vLLM measured, only 20.4–38.2% of KV-cache memory held actual token states, the rest lost to reservation for maximum length and fragmentation.¹⁸ Paging the KV cache in fixed blocks (PagedAttention) nearly eliminates the waste and gave 2–4× throughput at equal latency.¹⁸ More sequences per batch is the whole game in the memory-bound regime.

A latency floor

At batch 1, $\text{time per token} \ge W / \text{aggregate bandwidth}$. PaLM 540B with int8 weights on 64 TPU v4 chips gives $540\,\mathrm{GB} / (64 \times 1.2\,\mathrm{TB/s}) \approx 7\,\mathrm{ms}$. The measured low-batch latency was 29 ms per token.¹⁷¹³ The gap is the part of the problem a single-chip model leaves out: chip-to-chip communication (which shrinks slowly or not at all as chips are added), KV traffic, and sustained bandwidth below peak.¹⁷

The model is the one in “Why decode is memory-bound”: $\text{bytes} = W + BLk$, $\text{FLOPs} = B\,(2N + 4\, n_{\mathrm{layers}} L\, d_{\mathrm{model}})$, step = max of the two times, ideal overlap and 100% of peak. Peak compute doubles per halving of the weight format. Layer counts follow open dense models; edit layers, KV heads, head dimension and KV-cache format directly. Things to check:

• The knee $B^*$ moves with the ridge point and with $k$; note that changing the format moves it only when the KV format is decoupled from the weight format.
• Set KV heads to 64 (multi-head) on the default model and watch $B_{\max}$ and the knee.
• At 32K context, intensity levels off at one sequence’s FLOPs per KV byte, far below the ridge however large the batch. It tends toward $2\,(h/g) / b_{\mathrm{kv}}$ only as context grows further; quantizing the KV cache to 2 bits raises it.

KV-cache bytes per token at 16 bits for the Llama 3 family, from Table 3 of the paper (2 × layers × 8 KV heads × 128 × 2 bytes): 8B, 128 KiB; 70B, 320 KiB; 405B, 504 KiB.²⁰ At a 128K-token context the 70B model’s cache is 40 GiB per sequence, about a third of its 16-bit weights.

What each lever actually moves

Failure modes and fine print

• Format and the ridge. If a chip doubles peak FLOPS at each halving of precision and the KV cache follows the weights, the knee batch does not move: bytes and FLOP time shrink together. Quantization buys per-user speed and capacity, not a lower knee.
• Weight-only quantization is a memory optimization. If weights are dequantized to 16 bits for the math, compute time stays at the 16-bit rate, which only matters once batching has used up the memory-bound headroom.²⁵
• Speculation competes with batching for the same idle compute. At large batch the compute is no longer free, and the draft model’s own weights and KV cache take bandwidth and capacity.²⁴
• MQA and GQA change the model. MQA brings some quality loss; converting an existing multi-head checkpoint to GQA takes about 5% of its original pre-training compute.²¹²²
• Partitioning. Spreading a model across chips divides $W$ per chip but adds communication that grows in relative importance as chip count rises; for lowest latency the model is partitioned as far as it profitably can be, and small batches then cost more per token.¹⁷ See Scale-up fabrics.
• Computing in memory. The radical way around the wall is to stop moving weights at all, covered in Sparsity, in-memory and analog compute.

The decode model, step by step

Notation: $N$ parameters, $b_{\mathrm{w}}$ bytes per weight, $n_{\mathrm{L}}$ layers, model width $d$, $h$ query heads and $g$ KV heads of size $d_{\mathrm{h}}$ (so $d = h\, d_{\mathrm{h}}$), $b_{\mathrm{kv}}$ bytes per KV element, batch $B$, context $L$, bandwidth $\mathrm{BW}$, peak $P$ (FLOP/s at the chosen format), capacity $C$. This is the model in the simulation and, with chip count set to 1, the cost model of Pope et al.¹⁷

1. Weight bytes: $W = N b_{\mathrm{w}}$.
2. KV bytes per token: $k = 2\, n_{\mathrm{L}}\, g\, d_{\mathrm{h}}\, b_{\mathrm{kv}}$; per sequence, $K = Lk$.
3. Bytes per step: $M(B) = W + BK$.
4. FLOPs per token: $F = 2N + 4\, n_{\mathrm{L}} L d$ (dense matmuls plus the two attention products).
5. Step time: $t(B) = \max\!\left(M(B)/\mathrm{BW},\ BF/P\right)$. Per-user rate $1/t$, aggregate $B/t$.
6. Capacity: $W + BK \le C$, so $B_{\max} = \lfloor (C - W)/K \rfloor$.

The knee

Set memory time equal to compute time:

Solving for $B$:

• With $K \to 0$ and $F \approx 2N$: $B^* \approx b_{\mathrm{w}} (P/\mathrm{BW}) / 2$, the ridge point times bytes per weight over two. For a ridge of 250 FLOP/B and 16-bit weights, $B^* \approx 250$; at 8-bit with doubled $P$ (ridge 500), still 250.
• The denominator is the bytes a sequence “earns” per step ($F \cdot \mathrm{BW}/P$) minus the bytes it costs ($K$). When $K \ge F \cdot \mathrm{BW}/P$, there is no knee: decode is memory-bound at every batch size. With $F \approx 2N$ and $P/\mathrm{BW} = R$, the condition is $Lk \ge 2N/R$. For a 70B model with 16-bit GQA cache ($k = 327{,}680\,\mathrm{B}$) on a ridge-250 chip, that happens above about $L = 1{,}700$ tokens.
• The usable regime is $B \le \min(B^*, B_{\max})$. Above $B^*$, aggregate throughput saturates at $P/F$ tokens/s and per-user throughput falls as $1/B$; above $B_{\max}$, the batch doesn’t fit.

Worked example: the simulation’s defaults

70B parameters, 8-bit weights and KV, 80 layers, 8 KV heads × 128, $L = 512$, $\mathrm{BW} = 4\,\mathrm{TB/s}$, $P = 2{,}000\,\mathrm{TFLOPS}$ (8-bit), $C = 192\,\mathrm{GB}$:

• $W = 70\,\mathrm{GB}$; $k = 2 \times 80 \times 8 \times 128 \times 1 = 163{,}840\,\mathrm{B}$; $K = 84\,\mathrm{MB}$ per sequence.
• $B = 1$: memory time 17.5 ms, compute time 0.07 ms, so about 57 tokens/s. $B = 8$: 70.7 GB per step, 17.7 ms, about 57 tokens/s per user and 450 in total, math units busy 3% of the step.
• $B^* = 70\,\mathrm{GB} / (F \times 0.002\,\mathrm{B/FLOP} - 84\,\mathrm{MB}) \approx 352$; $B_{\max} = (192 - 70)\,\mathrm{GB} / 84\,\mathrm{MB} \approx 1{,}450$; the compute roof is about 14,100 tokens/s.

What the model leaves out

• Sustained versus peak. Real kernels reach a fraction of peak bandwidth and FLOPS, and memory and compute overlap imperfectly, so measured throughput is below both roofs and the knee is rounded, not sharp.
• Communication. With the model on $n$ chips, $W$, $K$ and $F$ per chip divide by $n$, but collective operations add time that shrinks slowly with $n$. The PaLM 540B example above (7 ms bound, 29 ms measured) shows how much that and other overheads can add.¹⁷
• Mixed phases. Real servers often batch prefill work together with decode steps, which raises the step’s intensity; or they separate the phases onto different machines.¹⁹
• Ragged contexts. Sequences in a batch have different lengths; $K$ is a sum over sequences, not $BL$. Allocator waste lowers the effective $B_{\max}$.¹⁸
• Activations and logits. Small compared with $W$ at modest batch, ignored here; the vocabulary projection is counted inside $N$.
• Mixture-of-experts. Only the routed experts’ weights do work per token, but across a large batch most experts can get used, so bytes per step can approach the full parameter count while FLOPs per token follow the active count. The model above would need separate $N$ for bytes and for FLOPs.

Roofline view

Dividing through by bytes gives the roofline form:³

$I(B)$ rises with $B$ and saturates at $F/K$, the per-sequence ceiling set by the KV cache. The knee exists only if $F/K$ exceeds the ridge $P/\mathrm{BW}$, which is the same condition as above. Every mitigation in the trade-offs table either raises $I$ (batching, smaller $W$, smaller $K$, speculation) or raises the slope $\mathrm{BW}$ (more or faster HBM, more chips, SRAM residency).