Multicore and vector units

Treat the chip as a power-limited system. You have a die full of transistors and a thermal budget, the TDP. Switching power is $P \approx \alpha C V^2 f$, and in the usual operating range the supply voltage must rise roughly in proportion to the clock, so one core’s power grows roughly as $f^3$. Once supply voltage stopped scaling with feature size, the cheapest way to turn transistors into throughput was more cores at moderate clocks, not one core at a higher one.²⁹

The rest of the chapter is about how that budget gets spent:

• Cores versus clock. With $P \propto N f^3$, a fixed budget gives $f \propto N^{-1/3}$ and peak throughput $Nf \propto N^{2/3}$, until the voltage floor, leakage and the uncore stop the gain. Amdahl’s serial fraction decides whether that throughput is usable.
• Threads per core. SMT trades per-thread latency for utilization of an expensiveout-of-order core.
• Data movement. Rings give way to meshes; the sliced last-level cache and memory controllers make latency depend on placement (NUMA); chiplets add die-to-die hops.
• Width per instruction. SIMD amortizes fetch, decode and scheduling over 4–16 lanes, from fixed-width (SSE/AVX/AVX-512, NEON) to length-agnostic (SVE, RVV) designs.
• Asymmetry. Big and little cores in one package, scheduled by capacity and energy.

Products appear only as case studies, x86 (AMD, Intel), Arm and RISC-V side by side, with dated figures in an As-of box.

In the range where a chip is designed to run, the supply needed for a clock $f$ rises roughly linearly with it, $V \approx V_0 + k f$, and switching power per core goes as $C V^2 f$. Take the sim’s illustrative curve (0.6 V up to 1 GHz, then +0.15 V per GHz): one core at 4 GHz needs 1.05 V and burns $1.05^2 \times 4 \approx 4.4$ units, while two cores at 2 GHz need 0.75 V and burn $2 \times 0.75^2 \times 2 \approx 2.3$ units, the same peak throughput for about half the power. That ratio is the economic argument for multicore, provided the work can be split.

The architecture-level evidence came before the device-level crisis. Agarwal et al. modeled an aggressive out-of-order core from 250 nm to 35 nm and found that, with clock gains shrinking and wire delay growing, no scaling strategy delivered more than 12.5% a year, against the 50–60% the industry was used to.⁴ The Berkeley report’s “brick wall” combined three limits: power, memory latency and ILP.² It also noted that static (leakage) power had reached as much as 40% of the total in desktops and servers, which matters for the multicore trade-off because idle cores still leak unless they are power-gated.²

Esmaeilzadeh et al. pushed the argument one step further. Combining device projections with multicore performance models, they estimated that 21% of a fixed-size chip would have to be dark at 22 nm and more than 50% at 8 nm, and that multicore scaling alone would deliver only 7.9× average speedup through 2024.⁹ Their exact percentages depend on the projections; the lasting result is that performance per watt, not transistors per die, sets the design. The responses all spend transistors to save energy: more and simpler cores, wider vector units that amortize instruction overhead, heterogeneous cores and fixed-function accelerators such as those in the AI chapters.

Even with perfect branch prediction and an unbounded window, IPC is bounded by the dataflow graph: total instructions divided by the critical-path length in cycles. Real cores sit well below that bound because the window is finite, mispredictions flush it and misses stall it. Tullsen et al. broke the 81% of wasted issue slots in their 8-issue model into many causes with no dominant one, which is why no single fix (a bigger predictor, a bigger cache) recovered it.⁶

The cost side grows superlinearly. Palacharla, Jouppi and Smith modeled rename, the wakeup and select logic that finds ready instructions, and the bypass network that forwards results: rename and wakeup delays have components quadratic in issue width, a fully bypassed design needs $2 \cdot IW^2 \cdot S$ bypass paths for issue width $IW$ and $S$ pipeline stages after the first result-producing stage, and the bypass wire delay grows quadratically with issue width. These sit on the critical path that sets the clock.⁵ Agarwal et al.’s wire-delay model is the quantitative version: as features shrink, structures that must be reached in one cycle cover a shrinking fraction of the die, so wide, deep cores lose clock rate or IPC, and annual gains from a conventional core fall to 12.5% or less.⁴

Two responses followed. Inside the core, simultaneous multithreading fills slots that one thread can’t (next sections). Across the chip, thread-level parallelism replaces instruction-level parallelism: the Berkeley report’s “new conventional wisdom” is that increasing parallelism, not clock frequency, is the primary way to raise processor performance.²

The two formulas measure $p$ on different machines, so they don’t contradict each other. Writing $s$ for the serial fraction on one core, the same program’s serial fraction measured on $N$ cores is $s' = s/(s + (1-s)/N)$; substituting it into Gustafson’s formula gives exactly Amdahl’s. What differs is the assumption about what is held fixed: work (Amdahl) or time (Gustafson).⁷

Hill and Marty made Amdahl’s law a design tool for multicore chips. Give the chip $n$ base-core equivalents (BCEs) of area and let a core built from $r$ BCEs run at $\mathrm{perf}(r) = \sqrt{r}$. A symmetric chip of $n/r$ such cores gives

and an asymmetric chip with one $r$-BCE core plus $n - r$ single-BCE cores runs the serial part on the big core and the parallel part on all of them. For $f = 0.975$ and $n = 256$ the best asymmetric design reaches 125.0 against 51.2 for the best symmetric one.⁸ That is the area argument for heterogeneous cores later in this chapter.

Power changes the axis. Hill and Marty count area; a power-limited chip also pays for frequency. Per-core power rising roughly as $f^3$ means a budget split across $N$ cores forces each to slow down, which costs the serial phase directly. Two hardware responses follow: boost clocks, which give one busy core the power that idle, switched-off cores aren’t using, and big cores dedicated to serial work. The sim below and Under the hood put numbers on both.¹⁰

What gets duplicated is small: architectural registers, PC, some control state and the interrupt controller per logical processor. Everything expensive, the execution engine, the caches and the bus interface, is shared between the threads.¹⁰ That is why SMT is cheap in area and why its benefit is largest on wide out-of-order cores with low single-thread utilization; the wider Zen 4 back end is AMD’s explanation for its higher SMT uplift.¹⁷

In Tullsen et al.’s terms, a miss creates vertical waste (a fully empty cycle) and limited ILP creates horizontal waste (a part-filled cycle). Fine-grained multithreading removes only vertical waste; SMT attacks both, and is “only limited by the issue bandwidth of the processor”.⁶ The costs are per-thread latency (a thread runs slower than alone), contention for shared caches and queues, which can make SMT a net loss for cache-sensitive code, and isolation: threads on one core share microarchitectural state, so some designs give each thread its own core. Intel cites “performance isolation” as a reason its E-cores are single-threaded.¹⁶

SMT also interacts with power. Two threads on one core use the same clock and voltage, so the extra throughput comes at nearly the cost of the extra switching it causes, without paying for a second core’s leakage and uncore share. That makes it one of the cheaper ways to raise throughput per watt when there are enough threads.

For a bidirectional ring of $N$ stops the average distance is about $N/4$ and the diameter $N/2$; for a $k \times k$ mesh with XY routing the average is about $2k/3$ and the diameter $2(k-1)$. At 64 cores that is 16 versus about 5 hops on average, which is why rings stopped at a couple of dozen stops. Intel’s Skylake-SP move to a mesh came with a cache rebalance: L2 grew from 256 KB to 1 MB per core and the shared LLC shrank from 2.5 MB (inclusive) to 1.375 MB (non-inclusive) per core,¹⁴ trading shared capacity for fewer trips across a larger network.

Slicing the LLC distributes both capacity and coherence work. Each address belongs to one home slice, and that slice’s snoop filter (a directory of sorts) tracks which private caches may hold the line, so a miss in one core’s L2 costs a mesh trip to the home slice and, if another core holds it modified, a forward from that core. The latency of a shared-cache hit therefore depends on where the home slice is, which makes even a single die somewhat non-uniform; sub-NUMA clustering restricts each half of the die to its own memory controllers and LLC slices to shorten those trips.¹⁴¹⁵ The coherence protocol itself (MESI-style states, snooping versus directories) and the ways software causes coherence traffic, such as two cores writing different variables in the same cache line, are in Caches and coherence.

Chiplet CPUs add a third tier of distance. With eight 8-core dies, only 7 of a core’s 63 peers share its die and cache; every other exchange crosses two die-to-die links and the I/O die. Putting the memory controllers on the central I/O die makes DRAM roughly equally far from every compute die, while core-to-core and cache-to-cache latency depends on which dies are involved.²⁴ Operating systems handle this with the same node abstraction and local-allocation policy they use for sockets.²⁰

A vector instruction amortizes the per-instruction overhead (fetch, decode, rename, scheduling, retirement) over 4–16 lanes, the same argument that drives matrix instructions in GPUs. AMD’s Zen 4 numbers make the power side concrete: AVX-512 cut the instruction count by up to half relative to AVX2, reduced front-end and out-of-order tracking power, and gave a 17% higher effective frequency on HPL in power-limited systems, even though most 512-bit operations take two passes through a 256-bit datapath.¹⁷ The flip side is that a wider unit switches more capacitance whenever it runs, so a chip sized for full-width vector code on every core must budget power for it, which is what the sim’s vector-width control models.

Masks and predicates

Conditionals and loop tails need per-lane control. AVX-512 adds eight opmask registers (k0–k7)¹⁰; SVE adds sixteen predicate registers and drives loop control from them, with whilelt building a predicate of the lanes still in range; it also adds first-fault loads, which let a loop read speculatively past the end of an array of unknown length.¹² RISC-V V uses register v0 as a mask and handles tails by shortening vl itself.¹³

Fixed width versus length-agnostic

• Binary portability. A length-agnostic binary runs on any implementation; a fixed-width one must be rebuilt (or multiversioned) to use wider units. That is why the x86 family has accumulated SSE, AVX, AVX-512 and AVX10 code paths.¹¹
• Tails and alignment. Predication or a shorter vl handles the last partial vector without a scalar remainder loop.¹²¹³
• What gets harder. The vector length is no longer a compile-time constant. Compilers can’t vectorize by unrolling a loop a fixed number of times, need instructions that use the current length implicitly (SVE’s index and inc), and must place spilled vector registers in stack regions whose offsets are multiples of the length.¹² RISC-V’s register grouping (LMUL) lets one instruction operate on 2, 4 or 8 registers at once, mainly so mixed-width data can be processed at the same element count.¹³

Under the hood walks through the same loop in SVE and RISC-V V assembly.

Hill and Marty’s asymmetric chip is the area argument: run the serial phase on a big core, worth $\sqrt{r}$ small cores of performance for $r$ of their area, and the parallel phase on everything.⁸ Kumar et al. made the power argument with real core designs at one ISA, picking the core per program phase.²³ Production systems need three things the models assume away:

• A scheduler that knows the cores. Linux’s capacity-aware scheduling uses capacity = work per Hz × max frequency to keep tasks that need more than a little core can give off the little cores, and energy-aware scheduling uses a per-core energy model to place each task.²¹²²
• ISA parity. A thread can migrate at any time, so every core must accept every instruction the thread might issue. Intel’s AVX10 specification makes the converged vector ISA a feature of both P-cores and E-cores.¹¹ The RISC-V V specification notes that a thread with live vector state cannot, in general, migrate between harts whose VLEN differs, so mixed designs keep one architectural vector length even when their datapaths differ.¹³
• Shared structures. Big and little cores share the cache hierarchy and memory, so their coherence and interconnect latencies differ, and a little core’s cache misses can still occupy shared bandwidth.

Brand names for the same structure: Arm’s big.LITTLE²¹, Intel’s P-cores and E-cores¹¹; in servers, dense-core variants such as AMD’s Zen 4c¹⁷ and Intel’s E-core Xeons¹⁶ apply the same idea one chip at a time.

The model, also in the sim spec: $V = 0.6\,\mathrm{V}$ up to 1 GHz, then $+0.15\,\mathrm{V/GHz}$; per busy core $(1 + 0.25\,w/128)\,\mathrm{nF} \cdot V^2 f$ (the vector term only if the job has vector code), plus $0.3\,\mathrm{A} \cdot V$ of leakage per core and 15 W of uncore. Run time is Amdahl’s, split again into scalar and vector work, with vector work $w/32$ times faster. The Expert view adds the budget, a boost toggle (serial code runs on one core with the others power-gated, at the fastest clock that fits) and a Gustafson toggle (parallel work grows with $N$). Try:

• Mixed job: the best design is 12 cores at 4.8 GHz. Turn boost on and the best design jumps to 64 cores at 1.9 GHz, 15.7× instead of 11.8×. This is why server chips advertise both a base and a boost clock.²⁵
• With boost off, set the mixed job to 512-bit vectors at 12 cores and fit the clock: the vector units’ power pulls the whole chip down to 3.8 GHz and the speedup falls to about 10×. Wide vectors pay only when most of the work is vector code.
• Halve and double the budget (60 W, 240 W) for the mixed job. The best core count moves 6 → 12 → 24 while the clock barely changes. Extra power buys cores, not frequency.

Deliberately left out: memory bandwidth and caches, coherence and synchronization costs, SMT, per-core voltage domains and temperature-dependent leakage. Parallel work scales perfectly, so the curves are upper bounds.

Per-core power from the TDPs: about 500/192 ≈ 2.6 W per core for the 192-core part, 400/64 ≈ 6.3 W for the 64-core part, both including uncore.²⁵ The ratio, about 2.4× power per core for about 1.5× base clock (and 1.35× boost), is consistent with power rising much faster than linearly in frequency once voltage must rise with it. The denser part’s Zen 5c cores follow AMD’s compact-core approach: the same IPC and features at lower maximum clocks in less area, which the Zen 4c generation put at about 35% less core-plus-L2 area.¹⁷

Vector datapaths differ more than their ISAs suggest: full 512-bit AVX-512 on Zen 5,²⁵ 512-bit operations on a 256-bit datapath on Zen 4,¹⁷ and 4 × 128-bit SVE2 on Neoverse V2,²⁶¹⁸ where Arm’s manual gives the SVE vector length as 128 bits, so vector-length-agnostic code sees 128-bit vectors and gets width from issuing to four units.¹⁹ Measured, rather than peak, comparisons belong to Comparing chips.

• Power versus serial speed. A fixed TDP split across more cores lowers the all-core clock; boost clocks win back serial performance only while the other cores are idle. The 192-core and 64-core EPYC parts above are the two ends of that trade in one family.²⁵¹⁰
• Bandwidth per core. Delivered throughput is $\min(N, B/d)$ core-equivalents for memory bandwidth $B$ and per-core demand $d$: a roofline in core count. The 192-core EPYC above has 12 DDR5 channels, one for every 16 cores,²⁵ and the Berkeley report counted this memory wall as one of the three walls.² Intel’s recent platform answers with more channels and multiplexed DIMMs aimed at more bandwidth per thread.¹⁵ The same pressure, far stronger, shapes AI accelerators; see The memory wall.
• Interconnect scaling. Rings stop scaling at a few dozen stops; meshes scale further but make LLC hit latency a function of distance; chiplets add a die-to-die tier.¹⁴²⁴
• SMT. Gains depend on how much one thread leaves idle, and threads contend for caches and queues; some designs drop SMT for isolation and predictable performance.⁶¹⁶
• Vector width. Wider registers raise peak throughput per instruction; fixed widths fragment the software ecosystem, and length-agnostic ISAs move the cost into compilers.¹¹¹²
• Software maturity. A new vector extension is only useful once compilers and libraries target it, as the SG2042’s pre-standard RVV 0.7.1 showed.²⁷

1. A power-constrained throughput model

Take $N$ identical cores sharing one voltage and clock. Per-core power is switching plus leakage, and the chip adds a fixed uncore:

In the range where $V \propto f$, the switching term is $\propto N f^3$, so the largest feasible clock is $f^* \propto (P_{\max}/N)^{1/3}$ and peak throughput $N f^* \propto N^{2/3} P_{\max}^{1/3}$. Two things end the gain. Below the voltage floor, $V$ stops falling, power becomes linear in $f$, and $N f$ is capped at roughly $P_{\max}/(C V_{\min}^2)$; meanwhile each extra core adds leakage and needs a share of the uncore whether or not it is busy. So there is an optimum $N$ for a given budget even for perfectly parallel work, and it rises with the budget. In the sim (Expert view, mixed job) the best core count goes 6 → 12 → 24 as the budget goes 60 → 120 → 240 W, at almost the same clock.

Now add Amdahl. With serial fraction $1 - p$ run on one core at $f_s$ and the rest on $N$ cores at $f^*(N)$:

where $f^*_1$ is the clock one busy core can reach with the others power-gated. Without boost, raising $N$ lowers $f_s$ and the serial term grows, which is why “a few fast cores” wins for serial code. With boost, $f_s$ is decoupled from $N$, and the optimum moves to many cores. That is the hardware form of Hill and Marty’s asymmetric chip, achieved in time (one core fast while the others sleep) rather than in area.⁸¹⁰

2. Hill and Marty’s three chips

With $n$ BCEs and $\mathrm{perf}(r) = \sqrt{r}$, the symmetric, asymmetric and dynamic speedups are⁸

The dynamic chip, which can gang $r$ BCEs into one core for serial code and split them for parallel code, bounds the other two. Boost clocks are a partial, power-based version of it: they can’t make a core wider, but they can give it the power that the idle cores aren’t using.

3. The same loop, length-agnostic

DAXPY, $y_i \leftarrow a x_i + y_i$, in SVE, from the SVE paper. There is no remainder loop and no constant for the vector length anywhere in the code.¹²

// x0 = &x[0], x1 = &y[0], x2 = &a, x3 = &n
daxpy_:
    ldrsw  x3, [x3]                    // x3 = *n
    mov    x4, #0                      // x4 = i = 0
    whilelt p0.d, x4, x3               // p0 = lanes with i+k < n
    ld1rd  z0.d, p0/z, [x2]            // broadcast *a
.loop:
    ld1d   z1.d, p0/z, [x0, x4, lsl #3] // x[i..]
    ld1d   z2.d, p0/z, [x1, x4, lsl #3] // y[i..]
    fmla   z2.d, p0/m, z1.d, z0.d       // y += a*x, active lanes only
    st1d   z2.d, p0, [x1, x4, lsl #3]   // store active lanes
    incd   x4                           // i += VL/64
.latch:
    whilelt p0.d, x4, x3               // recompute the predicate
    b.first .loop                      // loop while any lane is active
    ret

RISC-V V expresses the same idea with vsetvli, which returns how many elements the hardware will process this time; the loop subtracts that from the remaining count. From the specification’s vector-add example:¹³

# z[i] = x[i] + y[i], 32-bit integers; a0 = n, a1 = x, a2 = y, a3 = z
vvaddint32:
    vsetvli t0, a0, e32, ta, ma  # t0 = elements this pass (<= n)
    vle32.v v0, (a1)             # load x
      sub a0, a0, t0             # n -= t0
      slli t0, t0, 2             # bytes = t0 * 4
      add a1, a1, t0             # bump x pointer
    vle32.v v1, (a2)             # load y
      add a2, a2, t0             # bump y pointer
    vadd.vv v2, v0, v1           # z = x + y
    vse32.v v2, (a3)             # store z
      add a3, a3, t0             # bump z pointer
      bnez a0, vvaddint32        # loop while elements remain
      ret

Length-agnostic SIMD and SIMT converge from opposite ends. SVE and RVV take the width out of the binary but keep the vector in the program; SIMT keeps a scalar-thread program and puts width, masks and reconvergence in hardware, so correctness never depends on warp size while performance does.¹²¹³²⁸ The GPU then multiplies this chapter’s other levers: many simple cores per die, dozens of resident warps per core instead of SMT’s two to four threads, and matrix instructions that amortize per-instruction overhead even further than wide vectors. Continue with SIMT and GPUs; for how CPUs and accelerators are joined in a server, see Board and server.