Wafer-scale and SRAM-heavy designs

You know a neural network runs mostly on matrix multiplication. The chips in this chapter start from the observation that for many of those multiplications, especially in inference, the limit is moving weights and activations rather than doing arithmetic, and that moving data across a package or a board costs, by one vendor’s estimates, one to two orders of magnitude more energy per bit than moving it across a die.²³ Two design families follow:

• Wafer-scale integration stitches dozens of reticle fields into one part with wiring across the scribe lines, and survives defects with fine-grained redundancy. Cerebras’s WSE-3 has 4 trillion transistors, 900,000 cores and 44 GB of SRAM on one wafer.³ Tesla’s Dojo training tile (as presented at Hot Chips 2022) and a UCLA/UIUC prototype took the chiplet route instead, bonding known-good dies onto a wafer-sized substrate.¹⁰¹¹
• SRAM-heavy, reticle-sized chips, such as Groq’s tensor streaming processor (220 MB SRAM) and Graphcore’s IPU (896 MiB), keep the working set on-die and hand scheduling to the compiler.⁶⁷

The chapter works through the engineering problems in order: the reticle and the cost of off-die links, yield with redundancy, cross-reticle wiring, power delivery and cooling, keeping weights in SRAM, deterministic execution, and the capacity ceiling that pushes all of these designs back to multi-chip systems. Yield statistics are derived in the Transistors fabrication chapter; Under the hood extends them to repairable parts.

The scanner field is 26 × 33 mm (858 mm²) today and 26 × 16.5 mm with high-NA anamorphic optics.¹⁴ A 46,225 mm² wafer-scale part is about 54 full fields. Each WSE-2 die is 17 × 30 mm, within one field, and is repeated in a 12 × 7 grid of 84 dies, each a 66 × 154 array of cores (the slide lists 10,156 cores per die).²

The motivation is the gap between on-die and off-die interconnect. Cerebras’s own comparison, on an equal 826 mm² of silicon (their estimate of a GPU’s links, not a measurement), is:²

The 2024 version of the comparison puts each WSE-3 die’s edge links at 2,880 GB/s counting both directions (480 parallel 24 Gb/s lanes), at 0.05 pJ/bit per end, against 5 pJ/bit per end for 100 Gb/s SerDes.³ The reason is physics, not cleverness: a sub-millimeter parallel wire needs no equalization, clock recovery or long-reach driver, so it costs about what any on-chip wire costs. The cross-scribe interface is source-synchronous and parallel, with redundancy, link training and an auto-correction state machine.²

The alternatives trade differently. Chiplets on a silicon interconnect substrate keep links short (the UCLA/UIUC system uses 10 µm copper-pillar pitch and 200–300 µm links at 0.063 pJ/bit) and let every die be tested before assembly, but add bonding yield to worry about.¹¹ Tesla’s Dojo tile, as presented at Hot Chips 2022, combined 25 known-good compute dies and 40 I/O dies on a system-on-wafer substrate.¹⁰

Without redundancy, a 400 cm² monolithic part at $D_0 = 0.1\,\mathrm{cm^{-2}}$ has an expected 40 defects and Poisson yield $e^{-40}$. Negative binomial clustering helps $(\alpha = 1 \text{ gives } 1/41 \approx 2.4\%)$, but that is still not a product.¹⁵ The requirement becomes: every defect must land on something that can be replaced, and there must be a replacement within reach.

Three properties make that work:

• Granularity. Area lost per defect $\approx$ the replaceable unit’s area. WSE-2’s core is 228 × 170 µm (0.039 mm²) with 48 kB SRAM and half its area in memory.² WSE-3’s is about 0.05 mm², so Cerebras’s 46 expected defects remove about 2.2 mm².⁴
• Locality. A spare can only replace a core it can be wired in for, and the repaired network must still look like a complete rectangular mesh to software. The sim models this as independent repair regions, each with its own spares. That is why spare counts are set by the tail of the per-region defect distribution, not by the wafer-wide mean (see Under the hood).
• Everything is redundant. Cerebras provisions redundant fabric links as well as cores.¹ Any non-repairable area $A_{\mathrm{nr}}$ multiplies yield by $e^{-D_0 A_{\mathrm{nr}}}$: just 1% of a 400 cm² part (4 cm²) left unprotected costs a third of all wafers at $D_0 = 0.1$.

Cerebras’s numbers for WSE-3: 970,000 physical cores, 900,000 enabled (about 7% spare; Cerebras puts silicon utilization at 93%), against an expected ~46 defects at 0.001/mm².⁴ The ratio of spares to expected defects (about 1,500:1) is a reminder that locality and clustering, not the mean, set the budget; the blog does not say how the spares are placed.

The chiplet route moves the problem. In the UCLA/UIUC prototype, each chiplet is tested before bonding, so die defects are screened out, but each of its ~2,000 copper-pillar I/Os can fail to bond. With one pillar per pad (≥ 99.99% each) a chiplet bonds correctly only 81.46% of the time, which across 2,048 chiplets means about 380 bad ones. Two pillars per pad raise that to 99.998% and about one bad chiplet per wafer.¹¹ The network then has to survive the few that remain: a single dimension-ordered mesh lost over 12% of source–destination paths with five dead chiplets, and two independent meshes (X-then-Y and Y-then-X) cut that below 2%. The clock is forwarded tile to tile from any edge tile, so it reaches every working tile unless all four of its neighbors have failed.¹¹

Cross-reticle wiring

Scribe lines are normally keep-out: they are destroyed at dicing and carry the fab’s test structures. A monolithic wafer-scale part needs the foundry to allow product wiring through them; Cerebras did this with TSMC at 16 nm and extended it to 7 nm and 5 nm.¹²³ The link is designed like a very short parallel bus, not a SerDes: under 1 mm, source-synchronous, with redundancy, link training and an auto-correction state machine.² The result, per Cerebras, is the same bandwidth between dies as within one, so the compiler sees one uniform mesh.²

Power delivery

The numbers that matter are current and distance. At a supply around 0.8 V (an illustrative assumption, not a published figure), Dojo’s 15 kW tile would draw on the order of 19,000 A. Distributing that laterally through thin on-wafer metal is hopeless, because for a uniformly loaded region fed at its edge the IR drop grows with the square of its size (derived in Under the hood). Cerebras delivers, and the 2022 Dojo tile delivered, power vertically through the face of the part, so each region’s current path is short regardless of the wafer’s size.¹⁹

The UCLA/UIUC design quantifies the edge-fed alternative at much lower power. Its 1,024 tiles draw up to 350 mW each at 1.21 V, about 290 A in total, through two 2 µm-thick metal layers of the interconnect wafer. Feeding 2.5 V at the edge leaves about 1.4 V at the center under full load, so each chiplet has an LDO tracking a 1.4–2.5 V input to deliver 1.0–1.2 V.¹¹ The costs are visible: LDO efficiency is roughly $V_{\mathrm{out}}/V_{\mathrm{in}}$, so edge tiles waste more than half their input power; and because off-chip capacitors can only sit at the wafer’s edge, up to 70 mm away, about 35% of each tile’s area went to on-chip decoupling capacitance (about 20 nF per tile).¹¹ Vias through the full wafer thickness were the alternative, but were not yet available for the prototype.

Clocks

A single passive clock net across 15,100 mm² and 1,024 sinks has more than 450 pF and 120 nH of parasitics, limiting it to sub-MHz. The UCLA/UIUC design instead generates a fast clock in an edge tile and forwards it tile to tile, inverting it at each hop so duty-cycle distortion alternates instead of accumulating.¹¹ Graphcore runs its 823 mm² die from a mesochronous clock with about three cycles of global drift.⁷

Heat and mechanics

Heat flux, not total power, sets the cooling problem. Dojo’s 15 kW over 25 D1 dies of 645 mm² each is about 93 W/cm² of compute silicon.⁹ CS-1 drew 20 kW as a whole system;⁵ even if all of it went through the 462 cm² wafer, that would be about 43 W/cm². Cerebras described its wafer’s heat density as too high for direct air cooling and moved the heat into water through a cold plate.¹ Strain from thermal-expansion mismatch grows with distance from the part’s center; Cerebras judged that a traditional package would put too much mechanical stress on a wafer and developed a custom connector between wafer and board that absorbs the movement.¹

For a matrix–vector product, as in single-stream decode, each weight is used once per token, so arithmetic intensity is about one multiply-accumulate per weight read: the roofline puts it firmly on the bandwidth slope. Cerebras’s slides put the bytes-per-FLOP needed at about 0.005 for GEMM, 1 for GEMV and 2–3 for vector operations, and present distributed SRAM, next to every datapath, as the way to supply that ratio across a whole chip.²

Several of these designs still pair SRAM with DRAM for capacity. Dojo’s interface processors carried 32 GB of HBM each, 160 GB per tile edge, and a full training matrix had 1.3 TB of SRAM next to 13 TB of DRAM.¹⁰ Graphcore’s chips use host DDR as their off-die memory.⁷ SRAM is the working store; DRAM remains the warehouse.

Groq’s tensor streaming processor is the cleanest published example. The chip is functionally sliced: columns of memory, vector, matrix and data-reshaping units, with instructions flowing vertically to 144 independent instruction queues and operands flowing horizontally on eastward and westward stream-register paths.⁶ The design rules:

• No arbiters, crossbars, replay or caches; a defined memory consistency model with no reordering.
• The compiler tracks every tensor’s location and schedules at cycle granularity.
• A stream register is one hop per cycle, so travel time is a subtraction.
• NOP, SYNC and NOTIFY instructions let the compiler align the 144 queues; the result can be seen as a fully pipelined 144-wide VLIW machine.

Peak is about 750 TOPS of INT8 at 900 MHz from four 320 × 320 matrix engines, and instruction dispatch costs under 3% of area.⁶

Determinism is harder across chips. Link latency varies, there is no shared clock, and clocks drift. Groq gives each chip a free-running hardware-aligned counter, deskews links at run time, paces links in software so they never overflow or underflow, and uses forward error correction, which corrects most transmission errors deterministically. All on-chip SRAM and datapaths carry SECDED error-correcting codes. The chip-to-chip network is then software-scheduled, with no hardware flow control.⁶

Graphcore’s BSP model relaxes determinism to the superstep: tiles run asynchronously within a compute phase, then a hardware barrier (about 150 cycles on chip, 15 ns per hop between chips) precedes an exchange that the compiler schedules cycle by cycle, with addressing “by time and select state” over a 1,600-way receive multiplexer per tile.⁷ A superstep costs $\max_i(\text{compute}_i) + \text{exchange} + \text{sync}$, so load imbalance between tiles is paid every step. Graphcore also noted that bulk synchrony makes it harder to tune out supply-voltage margin for supply transients (all tiles switch phase together, so their current steps coincide).⁷

Cerebras sits between the two: 24 statically configured routes (“colors”) per link, each with its own buffering, so the network is non-blocking and its paths fixed at compile time, while cores fire handlers when data arrives and skip zero operands.² The common thread is that the compiler, not run-time hardware, owns placement and routing. That is a natural fit for SRAM-only machines, where every access latency is fixed.

Weight footprint is $Pb$ bytes; usable memory per device is $M(1-r)$, with $r$ the share reserved for KV cache, activations, code and buffers. The device count is $\lceil Pb / M(1-r) \rceil$, and in a pipeline mapping it is also the minimum number of stages. Published data points:

• WSE-3 (44 GB): Llama 3.1-8B (16 GB at FP16) on one wafer; Llama 3.1-70B (140 GB) on four, 176 GB total, which leaves about 20% of the SRAM ($r \approx 0.2$) for everything else. Layers are mapped to wafer regions sized by their memory and compute needs, and the model’s weights and KV cache stay in the region’s SRAM.³
• Between wafers, only activations cross: Cerebras reports needing about 100 Gb/s of 1.2 Tb/s of I/O per system, and says its four-wafer 70B mapping needs four network hops through I/O of under 5 µs, under 1% of latency.³

The same arithmetic for a 220 MB device gives $\lceil 140/0.22 \rceil = 637$ devices for FP16 weights and 319 for FP8, before any $r$; a pipeline that deep has hundreds of chip-to-chip hops per token, which is why multi-chip determinism matters so much to that design.⁶

Training changes the ratio. With mixed-precision Adam, model state is about 16 bytes per parameter (FP16 weights and gradients plus 12 bytes of FP32 master weights and two moments),¹² so a 3B-parameter model’s state alone (48 GB) would overflow a 44 GB wafer. Cerebras’s answer is weight streaming: an external store (MemoryX) holds weights and runs the optimizer; weights are broadcast onto the wafer one layer at a time; gradients stream out and are reduced on the way. Scaling to many systems is then data-parallel only.² This swaps the capacity limit for an external-bandwidth requirement, and only works because a training batch provides enough reuse per streamed weight.

Tesla’s system shows the same pressure from the other side: a training matrix with 1.3 TB of SRAM is paired with 13 TB of high-bandwidth DRAM, and the software keeps a single copy of parameters, replicating them just in time.¹⁰

Panel 1 shows one deterministic sample wafer (defects placed by a fixed seed) and, alongside, the model’s expectation: Poisson or negative binomial ($\alpha = 1$) die yield for the conventional dies, and $[P(k \le s)]^{400} \times (1 - s/2000)$ for the wafer-scale part. The Expert view adds $\lambda$ per die and per region and the probability that every region is covered. Panel 2 adds an HBM capacity slider and a reserve for KV cache and activations, and converts the SRAM needed into silicon area at about 1 MB/mm². Try:

• At $D_0 = 0.1$, how many spares per region give 99% part yield under Poisson? Switch to clustered: how many now? Why does clustering help the dies and hurt the wafer?
• At $D_0 = 1.0$, compare the spares needed with the expected defects per region.
• Set a 405B model at FP8 with a 20% reserve. How many wafers of SRAM is that?

The model assumes each defect kills exactly one core and that links, routers and clocks never fail, which makes it optimistic; see Under the hood for what that leaves out.

Ratios worth computing from the table:

• SRAM per area: about 0.95 MB/mm² (WSE-3), 1.1 MB/mm² (GC200), 0.68 MB/mm² (D1), logic included.
• Core size: 46,225 mm² ÷ 970,000 physical cores ≈ 0.048 mm² per core including everything else on the wafer, consistent with the 0.05 mm² Cerebras quotes.⁴
• Generational scaling at fixed area: WSE → WSE-3 (16 nm → 5 nm) is 3.3× the transistors but 2.25× the cores and 2.4× the SRAM, so each core grew (about 1.5× the transistors per core) while its SRAM stayed at 48 kB.¹³ The area cost of SRAM at advanced nodes is covered in The memory wall.
• Bandwidth per capacity: WSE-3 offers about 0.48 PB/s per GB of SRAM, Groq about 0.36 PB/s per GB. A GPU with 3.35 TB/s of HBM bandwidth over tens of GB is at about 0.0001 PB/s per GB or less, three to four orders of magnitude lower.³

Defect arithmetic, from the vendor: ~46 defects per wafer at 0.001/mm², costing ~2.2 mm², against 70,000 spare cores.⁴ CS-1 drew 20 kW as a system;⁵ a Dojo tile was specified at 15 kW.⁹

Capacity, bandwidth and batch

SRAM-resident designs win on bandwidth per byte stored, which is what single-stream decode needs. At high batch, HBM devices reuse each weight across many sequences, raise arithmetic intensity and close the gap, while the SRAM design must spend capacity on each sequence’s KV cache. Cerebras’s pitch is the low-latency end of that curve; it argues that surplus bandwidth also lets several users share the pipeline.³ The Comparing chips chapter discusses how to compare such claims on equal terms.

Cost structure

A wafer-scale part converts die-yield risk into spare area plus a small probability of losing a whole wafer. Wafer-scale integration has long been pitched as saving the cost of packaging and testing many separate chips,¹³ but it adds a custom package, connector and cooling of its own.¹ Cost per good die (wafer cost ÷ good dies) for conventional parts grows much faster than area;¹⁶ for a repairable wafer it is roughly wafer cost ÷ P(part works), with the spare fraction as an additional tax on delivered compute.

Failure modes

• Non-repairable critical area. Yield is multiplied by $e^{-D_0 A_{\mathrm{nr}}}$; 4 cm² unprotected on a 400 cm² part costs a third of wafers at 0.1/cm².
• Clustered defects exhaust local spares even when the mean is low; spare budgets are set by the tail.¹⁵
• Assembly yield. For chiplet-on-wafer, bonding defects at thousands of I/Os per die can dominate unless pads are duplicated.¹¹
• Supply and clock distribution. Edge-fed supplies droop quadratically with size; passive clock nets become too slow; both push toward local regulation and forwarded clocks.¹¹
• Synchronized power transients. Bulk-synchronous execution moves all tiles between phases together, complicating voltage-margin reduction.⁷
• Cross-chip determinism. Link latency variation and clock drift must be compensated, and link errors corrected in a way that doesn’t disturb the schedule.⁶

Four models behind the chapter and the simulation: yield with regional repair, the cost of non-repairable area, the IR drop of an edge-fed wafer, and capacity-driven device counts.

1. Die yield, briefly

With defects falling independently at density $D_0$, the number in area $A$ is Poisson with mean $\lambda = D_0 A$, so

and the zero-defect yield is $e^{-\lambda}$. If $D_0$ itself varies with a gamma distribution, the count is negative binomial with cluster parameter $\alpha$ and

$\alpha \ge 10$ is close to Poisson, $\alpha = 1$ close to Seeds’ model.¹⁵ Derivations and the Murphy model are in the fabrication chapter.

2. Yield with regional repair

Split the part into $R$ independent repair regions, each with mean $\lambda_r = D_0 A_r$ defects and $s$ spare units that can replace any failed unit in the region. Assume each defect kills one unit and units are small enough that two defects rarely share one. Then:

For $R = 400$, $\lambda_r = 0.1$ (1 cm² regions at $D_0 = 0.1\,\mathrm{cm^{-2}}$), Poisson:

Three observations. First, the step from $s = 1$ to $s = 2$ is enormous: each extra spare cuts the per-region failure probability by roughly a factor $\lambda_r/(s+2)$, so part yield climbs very steeply once the tail is covered. Second, the spare tax here is tiny (2 of 2,000 cores is 0.1%). Third, the required $s$ grows with $\lambda_r$: at $D_0 = 0.5$, $s = 4$ gives 93% and $s = 5$ gives 99.4%; at $D_0 = 1.0$, $s = 7$ gives 99.6%.

Clustering changes the tail. For $\alpha = 1$ the negative binomial is geometric,

At $\lambda_r = 0.1$ that is $0.0909^{s+1}$: $s = 2$ gives $P(\text{part}) = 0.74$ (versus 0.94 Poisson), $s = 4$ gives 0.9975. The same clustering raises conventional die yield (a 6.25 cm² die goes from 54% to 62%), because defects that bunch together spare other dies. Repairable parts care about the worst region; conventional dies care about the typical one.

Delivered compute is $P(\text{part ok}) \times (1 - s/C)$ for $C$ units per region. Maximizing it over $s$ gives the spare budget; in practice the budget is set well above this simple optimum to cover what the model ignores: defects that kill several adjacent units, parametric failures (slow or leaky cores), test escapes, and failures in the repair logic itself. Cerebras’s ~7% spare cores against ~46 expected defects suggests real budgets are far more conservative than this model.⁴

3. Non-repairable area

Any area $A_{\mathrm{nr}}$ whose failure cannot be repaired multiplies part yield by $e^{-D_0 A_{\mathrm{nr}}}$. With $A = 400\,\mathrm{cm^2}$ and $D_0 = 0.1\,\mathrm{cm^{-2}}$:

• 1% unprotected (4 cm²): $\times\, e^{-0.4} = 0.67$.
• 0.1% unprotected (0.4 cm²): $\times\, e^{-0.04} = 0.96$.

That is why wafer-scale designs make links, routers and clock paths redundant too.¹¹¹ The UCLA/UIUC dual-mesh result is the network analogue: with one dimension-ordered mesh, a single dead chiplet cuts every path that routes through it, and five dead chiplets disconnected over 12% of pairs; a second mesh routed Y-then-X gives most pairs a second path and brought that under 2%.¹¹

4. IR drop on an edge-fed wafer

Model the power grid as a disc of radius $R$ with sheet resistance $\rho_s$ (Ω per square), drawing a uniform current density $j$ (A/m²), fed at its rim. The current crossing a circle of radius $r$ is everything drawn inside it, $I(r) = j\pi r^2$. Spread over the circumference $2\pi r$, the sheet current density is $jr/2$, so the field is $\rho_s jr/2$. Integrating from the center to the rim:

At fixed power density, drop grows as $R^2$: doubling the part’s radius quadruples the droop. Total current grows as $R^2$ too, so raising the edge voltage only helps until regulators and losses take over, which is the compromise the UCLA/UIUC prototype made (2.5 V at the edge, about 1.4 V at the center, LDOs to 1.1 V).¹¹ Vertical delivery replaces $R$ with the spacing between feed points, which is independent of the part’s size. That is the geometric argument for feeding power through the face, as Cerebras does and the 2022 Dojo tile did.¹⁹

5. Device count and pipeline depth

with $P$ parameters, $b$ bytes per parameter, $M$ memory per device and $r$ the reserve for KV cache, activations and buffers. In a pipeline mapping, $\text{devices} \ge \text{stages}$, and each token crosses $(\text{stages} - 1)$ boundaries. Two consequences:

• Per-token latency includes $(\text{stages} - 1) \times \text{hop latency}$, so deep SRAM-only pipelines need very fast, predictable links. Cerebras reports that its four-wafer 70B mapping needs four network hops through I/O of under 5 µs, under 1% of latency.³
• Inter-device bandwidth per token is one activation vector per boundary (hidden size × bytes), tiny next to the weights each stage reads locally, which is why pipeline splits suit capacity-limited SRAM designs.³

For training, $Pb$ is replaced by the full optimizer state (about 16 bytes per parameter with mixed-precision Adam), the problem Cerebras’s weight streaming addresses by keeping weights and the optimizer in an external store.²

6. What the simulation leaves out

• Defects that hit links, routers, clocks or I/O (it treats all area as repairable cores).
• Defects large enough to kill several cores, and parametric failures.
• Scribe lanes, edge exclusion and partial dies: both options use the same 200 × 200 mm square.
• Correlation between regions on the same wafer: the clustered case uses an independent negative binomial per region for the expectation, and a heuristic cluster pattern for the sample map.
• Memory for anything but weights, unless you set the reserve.