Packaging and chiplets

You know an AI accelerator is a big chip with fast memory. This chapter is about the layer between the die and the board, where most of the interesting system decisions now happen: how a design is partitioned into chiplets, how those dies are joined, and what the joins cost.

UCIe’s own list of key metrics is a good map of the chapter: bandwidth density (per mm of die edge and per mm²), which comes from data rate and bump pitch; energy per bit; latency; channel reach; reliability; and the cost difference between standard and advanced packaging.⁴ We’ll derive the first two from bump geometry, follow them through 2.5D, 3D and HBM, and finish with the yield and cost model that decides whether a split pays.

The yield statistics themselves are in Making them; bumping, sort and known-good-die test are in After tapeout. Why accelerators need this much memory bandwidth is the subject of The memory wall.

Four forces

1. Field size. The 26 × 33 mm scanner field caps a monolithic die at about 858 mm², and high-NA EUV’s anamorphic optics halve it to 26 × 16.5 mm.² Die sizes of server CPUs and GPUs have been approaching that cap.¹
2. Defect-limited yield. Reticle-scale dies are expensive because area raises the chance of one or more defects.¹ Splitting turns a single large Bernoulli trial into several small ones whose failures can be discarded independently, provided each die is tested before assembly.
3. Node economics. Analog PHYs and much of the I/O die don’t scale, so a 7 nm core die plus a 12 nm I/O die beats an all-7 nm part.¹ The same logic now puts compute on N5 and cache/I/O on N6 in a single accelerator.⁸
4. Reuse and NRE. One I/O die and one core die covered 16 to 64 cores; a monolithic line could have needed four tapeouts, each with its own mask set and physical design.¹

What it costs

Per-chiplet overhead is real silicon: replicated clocking, power management, test and debug, plus the die-to-die PHYs.¹ Feng and Ma’s cost model assumes about 10% of each chiplet’s area for die-to-die interfaces, following EPYC. With that, they find splitting can save up to half the die cost, but packaging took about 30% of the total in the 16-core case they modeled, and for mature nodes the D2D and packaging overheads ate most of the savings.³ Their summary rule: splitting pays when the cost of die defects exceeds the total cost the packaging adds, and two or three chiplets usually capture most of the benefit.³

Non-recurring cost pulls the other way. Every distinct die needs its own masks and sign-off, so for a single product at modest volume a monolithic die is often cheaper overall; chiplets win through reuse across many products.³ The opposite bet, a whole wafer as one die with redundancy instead of packaging, is in Wafer-scale.

Three substrates, three wiring densities

Feng and Ma summarize the ladder: organic substrate (lines and spaces above 10 µm, highest per-lane rates), fan-out redistribution layers (above 2 µm) and silicon interposers (above 0.4 µm, lowest per-lane rates but the highest pin counts).³ Die-to-die bandwidth is pins × rate, and finer wiring lets you escape more pins from a given edge. UCIe accordingly defines two physical profiles: a standard-package profile at 100–130 µm bump pitch with up to 25 mm of reach, and an advanced-package profile at 25–55 µm with up to 2 mm.⁴

Organic-substrate links are not obsolete. AMD’s first EPYC ran high-bandwidth SerDes links across the package substrate between its four dies, avoiding the cost of a silicon interposer.¹

Interposer versus bridge

A full interposer gives every die access to fine wiring, but all signal and power vias must traverse it, and its area grows with the whole die complex. An embedded bridge confines fine-pitch wiring to the die-to-die region and lets each link use its own bridge, while the rest of each die keeps coarse bumps straight to the substrate.⁶ The price is two bump pitches on one die: fine at the bridge, coarse everywhere else. Variants add TSVs to the bridge to feed vertical power to HBM.⁶

Bandwidth per millimeter, step by step

1. Bumps on a square grid of pitch $p$ give $1/p^2$ bumps per mm². At 45 µm that is $1/(0.045\,\mathrm{mm})^2 \approx 494$ per mm².
2. Not all are signals: power, ground, clocks, valid and sideband take a share. Call the signal share $f$.
3. Only a strip of depth $D$ behind the edge is used, so signals per mm of shoreline $= fD/p^2$.
4. Multiply by the lane rate $R$ and divide by 8 for bytes: shoreline density $= fDR/(8p^2)$.

UCIe’s own bump maps pin this down. An x64 advanced module occupies 388.8 µm of shoreline at every pitch; its depth is about 388 µm at 25 µm pitch (16 bump columns), 1,043 µm at 45 µm (10 columns) and 1,585 µm at 55 µm (8 columns).⁴ At 32 GT/s a module moves 256 GB/s each way, and 512 GB/s over 0.3888 mm is the 1,317 GB/s/mm at the top of the advanced range; a standard x16 module moves 64 GB/s each way.⁴ In the simulation, $f \approx 0.68$ and $D = 1\,\mathrm{mm}$ reproduce that figure at 45 µm.

Pitch scaling and what it does to the PHY

UCIe-3D is specified for bond pitches under 10 µm (and functional from 10 to 25 µm), with connection count scaling as $1/p^2$ over the entire chiplet area rather than its edges.⁵ Its targets by pitch:

Source: UCIe 2.0 presentation.⁵ Each step follows $1/p^2$: $(9/5)^2 \approx 3.2$, $(9/3)^2 = 9$, $(9/1)^2 = 81$. The data rate is capped at 4 GT/s, chosen to equal the SoC’s logic frequency, with 80 lanes per cluster.⁵ With thousands of bonds per mm², bandwidth comes from width rather than speed. Running lanes near the core clock removes serializers, most of the clocking and the equalization a lateral link needs, which is where the energy saving comes from. The same presentation lists electrostatic-discharge protection falling from 30 V to 5 V CDM, and possibly to none for wafer-to-wafer bonds, which removes more capacitance from every lane.⁵

Bonding flows

Recessed Cu pads in oxide, oxide activation, face-to-face oxide bonding, then an anneal (typically around 300 °C) in which copper expansion closes the gap.⁷ Wafer-on-wafer reaches 400 nm pitch in research with near-perfect planarity from CMP; chip-on-wafer has reached about 2 µm because singulated dies can’t be planarized the same way, but it permits known-good-die selection.⁷ Wafer-to-wafer therefore suits stacks of high-yield, same-size dies; chip-to-wafer suits large or immature logic dies.

Thermal coupling

Stacking roughly doubles power density per footprint when both tiers are active, and heat from the lower tier must cross the upper tier and the bond layer to reach the heat sink. In Mathur et al.’s calibrated simulation of a two-tier 7 nm CPU, overlapping hotspots raised peak temperature by up to 12 °C over 2D; placing logic over memory, so hotspots don’t overlap, cut that to about 6 °C.¹⁴ That is why the 3D products in this chapter pair compute with cache or I/O dies rather than stacking compute on compute.

The stack’s energy budget

O’Connor et al. break down an HBM2 access: about 1.21 pJ/bit of row activation, 2.24 pJ/bit to move data across the DRAM die, down the TSVs and across the base die (about 9.9 mm in all), and only 0.3 pJ/bit on the interposer I/O, for 3.92 pJ/bit including ECC.¹¹ The package link is the cheap part. At that energy a 60 W DRAM power budget caps bandwidth near 2 TB/s, and 4 TB/s would cost over 120 W.¹¹ The authors concluded that future high-bandwidth DRAM must cut this internal data-movement energy, not just add pins.¹¹

Reach and floorplan

HBM’s I/O is unterminated, wide and slow per pin, which only works over a short, low-loss channel. On an interposer, a copper line 1.5 µm thick, 1 µm wide and 2.5 µm from its neighbors supports about 5 mm at 2 Gb/s; the HBM2 PHY could reach roughly 5–7 mm.¹¹ The PHYs therefore sit on the compute die’s edge facing each stack, and HBM competes with die-to-die links and off-package I/O for the same shoreline. A larger compute die has more edge, and so room for more stacks.

Generations

Sources: JEDEC announcements¹²¹³; HBM3 width from AMD’s bandwidth arithmetic (8,192 bits across eight stacks).⁹ Doubling width per stack doubles the microbumps and interposer routing per stack, so each generation tightens the packaging as well as the DRAM. Taller stacks push toward hybrid bonding between DRAM dies: production stacks are 8 to 12 dies high, and 16-high hybrid-bonded HBM has been demonstrated.⁷

Physical layer

A module has 16 (standard) or 64 (advanced) single-ended data lanes in each direction, plus one valid lane, a differential forwarded clock and a track lane for calibration; a link is one, two or four modules. Supported rates are 4, 8, 12, 16, 24 and 32 GT/s, and a device must support every rate up to its maximum. At 32 GT/s that is 64 GB/s per standard module and 256 GB/s per advanced module, each direction. An always-on sideband (two lanes each way at 800 MHz) handles training and management using otherwise depopulated bumps.⁴ The bump maps are fixed in the specification, with die rotation and mirroring allowed, so dies from different vendors line up.⁴

Adapter and protocols

The die-to-die adapter builds 68-byte or 256-byte flits with a 2-byte header and CRC covering each 128-byte payload chunk, and replays on CRC failure. Raw mode bypasses the adapter for protocols that bring their own.⁴ PCIe and CXL map on for plug-and-play attach; streaming mode carries a vendor’s on-die fabric protocol (AXI, CHI and others), so a CPU or switch can be built from identical chiplets without protocol conversion.⁴

Targets

Latency under 2 ns transmit plus receive including adapter and PHY; low-power entry and exit in 0.5–1 ns; and FIT far below 1 with flit-mode CRC and retry.⁴ Repair differs by profile: spare lanes in advanced packages, width degradation in standard ones. Version 1.1 added per-lane error counters for run-time link-health monitoring (for automotive use) and a 32-lane advanced option that cuts PHY area by about 40% for lower-cost packaging.⁴ Version 2.0 added UCIe-3D and a DFx architecture for discovery, test, debug and telemetry across all chiplets in a package, from die sort to field.⁵

Many shipping multi-die products still use proprietary links tuned to one package: AMD’s Infinity Fabric and NVIDIA’s NV-HBI are examples (see By the numbers). The standard’s value is the open chiplet market it enables, more than any one link’s performance.

The link model is $N = fDL/p^2$ for side-by-side links (signal share $f$, bump-field depth $D$, shoreline $L$, pitch $p$) and $N = fA/p^2$ for a bonded overlap $A$. Bandwidth is $NR/8$ summed over both directions, and power is bandwidth × energy per bit. The signal shares (0.38, 0.68, 0.65) are calibrated so the defaults land near UCIe’s published densities. The lower panel compares silicon cost per good system for a monolithic die and a split into chiplets, with negative binomial yield. Try:

• In 2.5D, set 32 Gb/s and 45 µm and compare the shoreline density with UCIe’s 1,317 GB/s/mm. Then shrink the depth: what does a shallower PHY cost you?
• Find the logic area at which four chiplets stop beating a monolithic die at $D_0 = 0.1/\mathrm{cm}^2$. How does the crossover move with $D_0$ and with die-to-die overhead?
• Compare tested and untested chiplets at $D_0 = 0.3/\mathrm{cm}^2$. Why does the gap explode?

It leaves out signal integrity (reach is shown, not enforced), package and interposer cost, bonding yield per connection, test escapes and NRE. Use it for scaling, not for quotes.

Two cross-checks. MI300X’s HBM figure is plain arithmetic: $8 \times 1{,}024\,\mathrm{bits} \times 5.2\,\mathrm{Gb/s} \div 8 = 5.3\,\mathrm{TB/s}$.⁹ And NV-HBI’s 10 TB/s across one die boundary is in the range the simulation’s 2.5D model gives for a die edge of about 8–15 mm at 16–32 Gb/s and 45 µm pitch. NVIDIA doesn’t publish NV-HBI’s pitch or lane count, so that is a consistency check, not a reverse-engineering.

Where designs go wrong

• Over-splitting. Savings from finer granularity are marginal past two or three chiplets while D2D area and packaging overhead keep growing.³
• Advanced packaging on a mature node. At 14 nm-class defect densities, Feng and Ma find D2D and packaging overhead above 25% for organic multi-chip modules and above 50% for 2.5D, wiping out most of the yield gain; advanced packaging pays mainly with advanced logic.³
• Packaging yield and wasted known-good dies. With a large monolithic interposer, interposer yield and bonding defects scrap good dies; at 7 nm and 900 mm² in 2.5D, packaging cost was comparable to chip cost in their model.³
• Shoreline exhaustion. HBM PHYs, die-to-die links and off-package SerDes all compete for the same edge, and edge grows only as $\sqrt{\text{area}}$.⁴¹¹
• Generality against optimization. A common chiplet interface sized for the most demanding die makes every other die pay its area; a package provisioned for any chiplet combination is over-designed for di/dt, electromigration and decoupling on most of them.¹
• Stacking hot on hot. Overlapping hotspots cost up to 12 °C in a two-tier CPU study; thermally aware partitioning halved it.¹⁴
• Hybrid-bond process windows. Bonding needs near-perfect flatness and cleanliness; a warped wafer leaves regions unconnected, and wafer-on-wafer pairs dies regardless of whether they are good.⁷
• Coherence and security. Mixing chiplets from several sources raises open questions about coherence protocols, memory consistency, denial-of-service between chiplets and side channels on the shared fabric.¹

Three small models: connection count and bandwidth density from bump geometry, link power, and the silicon cost of a split versus a monolithic die. These are the equations behind the simulation.

1. Counting connections

For a square bump array of pitch $p$ (in mm) with a share $f$ of bumps carrying data:

A side-by-side link uses a strip of depth $D$ behind the shoreline $L$; a stacked link uses the whole overlap area $A$:

Bandwidth, counting both directions, is $\mathrm{BW} = NR/8$ in GB/s for a lane rate $R$ in Gb/s. Worked at the simulation’s 2.5D defaults ($f = 0.68$, $D = 1\,\mathrm{mm}$, $L = 8\,\mathrm{mm}$, $p = 45\,\mu\mathrm{m}$, $R = 16\,\mathrm{Gb/s}$):

1. $1/p^2 = 1/(0.045)^2 \approx 494$ bumps/mm²; $\times\, 0.68 \approx 336$ signals/mm².
2. $\times\, D\ (1\,\mathrm{mm}) \approx 336$ signals per mm of edge; $\times\, L\ (8\,\mathrm{mm}) \approx 2{,}690$ signals.
3. $\times\, 16\,\mathrm{Gb/s} \div 8 \approx 5.4\,\mathrm{TB/s}$, or about 670 GB/s per mm of shoreline.

The calibration check is UCIe’s own bump map: a 64-lane module (128 data lanes counting both directions) in 388.8 µm × 1,043 µm at 45 µm pitch carries 512 GB/s at 32 GT/s, which is 1,317 GB/s per mm of shoreline.⁴ The model gives 1,343 GB/s/mm at the same settings. For 3D, $f = 0.65$ at 9 µm and 4 GT/s gives $0.65 \times 12{,}346 \times 4/8 \approx 4{,}000\,\mathrm{GB/s/mm^2}$, UCIe-3D’s stated target.⁵

Two consequences fall straight out. Edge links scale as $D/p^2$, so halving pitch at fixed depth quadruples bandwidth; at fixed bump count it instead shrinks the PHY’s depth, which is how UCIe’s 388.8 µm module goes from 1,585 µm deep at 55 µm pitch to 388 µm at 25 µm.⁴ And 3D links scale with area, so a stacked die’s die-to-die bandwidth grows with its footprint while a side-by-side die’s grows only with its perimeter.

2. Power at bandwidth

One terabit per second at one picojoule per bit is one watt. The 5.4 TB/s link above at 0.25 pJ/b: $43\,\mathrm{Tb/s} \times 0.25\,\mathrm{pJ} = 10.7\,\mathrm{W}$. NV-HBI-class bandwidth of 10 TB/s at the same energy would be 20 W, under 3% of a 750 W accelerator such as MI300X.⁹ UCIe claims 20× the I/O performance at a twentieth of the power of off-package SerDes, which is why links this wide stay inside the package.⁴ The 3D numbers show why stacking wins on energy: 4 TB/s through 1 mm² at under 0.05 pJ/b is under 1.6 W.⁵

3. Silicon cost of a split

The simulation’s lower panel uses the standard textbook model:

1. Dies per wafer for area $A$ on a wafer of diameter $d$:¹⁵
   $$\mathrm{DPW} = \frac{\pi (d/2)^2}{A} - \frac{\pi d}{\sqrt{2A}}$$
2. Negative binomial yield with defect density $D_0$ and cluster parameter $\alpha$:¹⁵³
   $$Y = \left(1 + \frac{D_0 A}{\alpha}\right)^{-\alpha}, \qquad \alpha \approx 3$$
3. Cost per good die:¹⁵
   $$C = \frac{\text{wafer cost}}{\mathrm{DPW} \times Y}$$
4. Split into $n$ chiplets with die-to-die overhead $h$: each chiplet has area
   $$a = \frac{A}{n}(1 + h)$$
5. With known-good-die test and a bond yield $y_{\mathrm{b}}$ per chiplet:
   $$C_{\mathrm{split}} = \frac{n\, C(a)}{y_{\mathrm{b}}^{\,n}}$$
6. Without test, every package needs $n$ good dies:
   $$C_{\mathrm{untested}} = \frac{n \cdot \text{wafer cost} / \mathrm{DPW}(a)}{Y(a)^n \, y_{\mathrm{b}}^{\,n}}$$

Worked at the simulation defaults ($A = 800\,\mathrm{mm}^2$, $D_0 = 0.1/\mathrm{cm}^2$, $n = 4$, $h = 10\%$, $y_{\mathrm{b}} = 99\%$, a $10,000 illustrative 300 mm wafer):

• Monolithic: $\mathrm{DPW} = 64$, $Y = (1 + 0.8/3)^{-3} \approx 49\%$, so about $318 per good die.
• Chiplets: $a = 220\,\mathrm{mm}^2$, $\mathrm{DPW} = 276$, $Y \approx 81\%$, so about $45 per good chiplet; $\times\, 4 \div 0.99^4 \approx \$187$ per system.
• Untested: $0.81^4 \approx 43\%$ of packages work, so about $353, worse than monolithic.

So the tested split costs about 0.59× the monolithic silicon. Push the area down to 100 mm² and the ratio rises above 1: the overhead outweighs the yield gain. This is the curve behind Feng and Ma’s turning point.³

What the simple model leaves out, from Feng and Ma’s fuller version: the raw package and substrate cost; for 2.5D the interposer’s own cost and yield $y_1$; chip-to-interposer bond yield $y_2$ per chip and interposer-to-substrate bond yield $y_3$; and the good dies scrapped when packaging fails. Their packaging term is

and the last term, wasted known-good dies, is why chip-last assembly (build the interposer or redistribution layers first, then bond the dies) is preferred for expensive dies.³ NRE adds a per-die fixed cost (masks, sign-off) plus per-area terms, amortized over volume.³

4. Why stacking heats up

To first order, temperature rise is power times thermal resistance to the heat sink. Two tiers in one footprint double the power through the same sink area, and the lower tier’s heat adds the resistance of the upper die and the bond layer to its path. Overlapping hotspots compound both. Mathur et al. quantify it for a calibrated 7 nm two-tier CPU: up to 12 °C above the 2D design, about 6 °C with logic over memory.¹⁴ Every degree of headroom lost there comes out of clock frequency or sustained workload before throttling.