Architecture

Architecture defines the problem everyone downstream solves. It fixes:

• the partition: which blocks exist and how the design nests into sub-blocks;
• the memory hierarchy and the bandwidth between its levels;
• the interconnect, and how copies of data in different caches are kept consistent;
• how many pipeline stages each datapath has, and so how much logic must fit between registers in one clock cycle;
• the clock and power domains.

Those choices bound what later stages can reach. Synthesis maps the RTL onto gates, the floorplan places the big blocks on the die, and timing closure makes every path meet the clock. None of them can rescue a pipeline stage that holds too many levels of gates for one cycle, or a wide bus that the architecture says must cross the whole die in a single cycle. Fixing either means changing the RTL, and so the architecture.

The work is quantitative, and it runs in rounds. First, simple formulas prune the space of possible designs: Amdahl’s law (how much speeding up part of a job helps), the roofline model (whether a task is limited by compute or by memory bandwidth), AMAT (average memory latency) and queueing models (how waiting time grows with load). Each is explained further down this page. Second, cycle-level simulators rank the surviving designs on recorded workloads. Third, power and area models such as McPAT turn the simulator’s activity counts (how often each cache, core or link is used) into watts and square millimeters.⁶

Hardware and software are designed together. For many accelerators the algorithm itself changes, trading extra arithmetic, which is cheap in custom hardware, for less memory traffic, which is expensive; even then, memory usually ends up taking most of the accelerator’s area and power.¹ The Architectures guide follows that thread in The memory wall.

Energy gap, 4-core processor chip vs. custom ASIC, HD video encoding

500×

Energy to fetch and decode an instruction vs. one add

10–4000×

Share of the TPU v1 die spent on control logic

2%

An ASIC (application-specific integrated circuit) is a chip custom-built for one task. The figures come from Hameed et al., Dally et al. and Google’s paper on the TPU v1, a custom chip for running neural networks.²¹⁹

Two outputs are easy to underrate.

Interface definitions. For every connection the spec pins down the protocol (the rules for requesting and answering a transfer), the data width, how many requests may be outstanding at once, whether responses must come back in order, and the address map. These are what let blocks from different teams and suppliers work together. Get one wrong and it surfaces late, when the blocks are first connected.

Per-block budgets. Large chips are built hierarchically: each big block is synthesized, placed and routed on its own, then assembled. A block’s budget of area, power and cycles of latency becomes its timing budget (how much of each clock cycle its inputs and outputs may use) and the first input to the floorplan. Treat every budget as an estimate with an error bar and record the assumption behind it, so that when RTL numbers arrive you know which decisions to revisit.

Partition by traffic

A good partition keeps heavy, latency-critical traffic inside a block and puts traffic that can tolerate delay on block boundaries. Count the boundaries before you commit, because each one costs twice later. Every boundary becomes a boundary in hierarchical implementation, where the block is laid out on its own and needs its own timing budget and pin plan. Every boundary between unrelated clocks also becomes a clock-domain-crossing structure (a synchronizer or an asynchronous FIFO, a small buffer written on one clock and read on the other) that must be verified.

Memory system

AMAT is recursive: the L1 miss penalty is the L2’s own AMAT, and so on down to DRAM. The tension at L1 is capacity against hit time: a bigger cache misses less but answers more slowly, and every instruction that touches memory waits on it. One rule of thumb is to use the biggest L1 that keeps hit time at 1–2 cycles, about 16–64 KB in modern technology.¹⁰

Accelerators often trade caches for scratchpads. The TPU v1 used a large software-managed memory and a deterministic execution model, with no caches, branch prediction or out-of-order execution, because predictable timing matched its 99th-percentile latency limits better than CPU and GPU features that improve only the average case.⁹

Bandwidth also needs concurrency. Little’s law says the average number of requests in flight equals the request rate times how long each one takes.²² Turned around: to sustain bandwidth $B$ when each request takes latency $W$, a requester must keep $B \cdot W$ bytes outstanding. At 17 GB/s and 100 ns that is 1,700 bytes, about 27 cache lines of 64 bytes, and the hardware needs a tracking slot for each one: miss-handling registers in a cache, tags in a DMA engine, buffer entries in the interconnect.

Interconnect and coherence

Topology choice trades four things: hop count (routers a packet passes through, a proxy for latency and energy), bisection bandwidth (the total bandwidth across a cut that splits the network into two equal halves, a proxy for how much chip-wide traffic it can carry), router degree (ports per router, which sets router size) and how easily the links lay out on a flat die. Under uniform random traffic, average hop count runs ring > mesh > torus, and a mesh has half the bisection bandwidth of a torus of the same size.¹¹

Coherence choices drive interconnect traffic. A directory avoids broadcasts but adds storage and, when another cache holds the data, an extra trip through the directory.¹² Accelerators are often attached without coherence and move data with DMA (direct memory access, a block that copies data between memories without a processor’s help); the TPU v1 uses a programmable DMA controller between host memory and its Unified Buffer.⁹ Coherent attachment simplifies software but puts the accelerator’s traffic into the coherence protocol, right next to the CPU caches. The protocols, directory organizations and false sharing are worked through in Caches and coherence.

Specialization and its limits

Accelerator speedup is bounded three ways. The first is the offloadable fraction: Amdahl’s law, with the time to hand work over and synchronize counted in the part that isn’t sped up. The second is memory bandwidth, which the roofline model captures. The third is programmability: a block too narrow for next year’s algorithms becomes dead silicon. Dally et al. describe accelerator design as parallel programming guided by a cost model in which arithmetic is free and global memory is expensive.¹ The TPU v1 paper estimates that replacing its DDR3 memory with the faster GDDR5 used on GPUs of the time would have roughly tripled achieved throughput, a reminder that the memory interface is part of the accelerator.⁹ The Architectures guide develops these trade-offs for AI chips, starting from What the workload needs.

Clock and power domain strategy

Each power domain multiplies downstream work. Signals leaving a domain that can be off need isolation cells, which hold them at a fixed value so they don’t feed garbage into live logic. Signals between domains at different voltages need level shifters. Registers that must keep their contents through power-off need retention versions. The domain also needs power switches, a set of power states to verify, and an area of its own in the floorplan.⁷⁸

Unrelated (asynchronous) clock domains save power and let blocks run at their own rates, a style called GALS, globally asynchronous, locally synchronous. But every crossing needs structural and formal CDC checks, because neither RTL simulation nor static timing analysis (STA, which checks every path against the clock without simulating) catches metastability bugs.¹³ Fewer, well-chosen domains usually beat fine-grained ones whose savings disappear into overhead.

Make or buy, chiplets

Make-or-buy is a risk decision as much as a cost one. Protocol versions, verification quality, configurability, test hooks, power intent and license terms all land on the team that integrates the block. Generator-based IP makes reuse easier, because one generator produces many configurations.¹⁴

For chiplets, the partition must respect die-to-die bandwidth. The UCIe consortium, which publishes an open die-to-die standard, cites 20× the I/O performance at 1/20 the power of off-package SerDes (the high-speed serial links used between packages). Its targets are still not free: 0.25 to 0.5 pJ per bit and under 2 ns of added latency for transmit plus receive.¹⁵ Each crossing costs PHY area at the die edge, energy per bit and nanoseconds that an on-die wire avoids, so put chiplet cuts where traffic is thin.

Treat the models as a ladder trading speed for fidelity. Analytical models evaluate thousands of design points per second and catch first-order mistakes, such as a bandwidth shortfall or an Amdahl cap. gem5’s CPU models sit at deliberately different points on the speed-versus-accuracy spectrum; the simplest is suited to fast-forwarding, so one study can skip quickly through a program’s start-up and then measure the region of interest with the detailed model.³ RTL simulation and FPGA emulation (running the RTL on a reprogrammable chip) come later; they are the reference you calibrate the faster models against.

Roofline is most useful with ceilings: lower lines under the roof that show the limit when an optimization is missing, for example code that doesn’t use SIMD instructions or reads memory in a scattered order. The roofline authors measured sustainable memory bandwidth with their own microbenchmarks rather than trusting the datasheet peak.¹⁶ For the TPU v1, the ridge point (where the slope meets the flat roof, the intensity needed to reach peak) sat at 1,350 operations per byte of weight memory, against 13 for a contemporary server CPU and 9 for a GPU. Its measured MLP and LSTM layers, network types that use each weight only a few times, sat under the bandwidth slope.⁹

Early power models run off the same activity counts. McPAT reads per-component access counts from the performance simulator through an XML file and turns them into dynamic power (from switching) and leakage power (which flows even when nothing switches).⁶ Before RTL exists their absolute numbers carry wide error bars; use them to compare candidates, and re-anchor them to synthesis results as blocks mature.

The sim also shows the formula and the analytic optimum. Time per instruction is the clock period times CPI, $(2400/k + t_{\mathrm{o}})\bigl(1 + c\,(k - 1)\bigr)$, where $t_{\mathrm{o}}$ is the register overhead. Setting its derivative in $k$ to zero gives $k^* = \sqrt{2400 \times (1 - c) / (t_{\mathrm{o}}\, c)}$. That has the same square-root form as Hartstein and Puzak’s $p_{\mathrm{opt}}^2 = N_{\mathrm{I}}\, t_{\mathrm{p}} / (\alpha \gamma N_{\mathrm{H}}\, t_{\mathrm{o}})$: total logic delay over latch overhead in the numerator and a hazard term in the denominator (the symbols are explained under “Under the hood”).¹⁹ Sweep the inputs: 99% prediction accuracy moves the peak to $k = 20$, 80% moves it to $k = 15$ (exactly tied with 16), and 150 ps of overhead moves it to $k = 14$.

The flat top is the practical lesson. A few stages either side of the optimum cost under 1%, so real teams pick depth for power, verification effort and timing margin. Weighting power in the metric pushes the optimum shallower still.¹⁹

Real cores also shrink $c$ itself, with TAGE-class predictors and out-of-order windows that overlap stalls; see Branch prediction and out-of-order execution.

Below are an illustrative block-level summary, a back-of-the-envelope model run against it, and a trade-off table of the kind an architecture review works through. The numbers are invented but internally consistent; check that the model’s conclusions follow from the summary.

The architecture’s cycle budgets are physical claims. A link that must cross a 10 mm die in one cycle, a cache sized for a 1-cycle hit, or a stage holding a certain number of FO4 delays all assume wire and gate delays that the floorplan must then deliver. (One FO4 is the delay of an inverter driving four copies of itself, a unit that stays roughly constant across manufacturing processes.) Hrishikesh et al. put the best clock period for integer code at about 8 FO4: 6 of useful logic and about 2 of latch and clock overhead. They also found that further pipelining could at best double integer performance over the designs of the day.²⁰

At that depth, overhead is a quarter of the cycle, so the clock tree’s skew (the clock arriving at different flip-flops at slightly different times) and jitter (variation from one tick to the next) matter as much as the logic. Budget global links in cycles from the start. Adding pipeline registers to a link whose protocol assumed one-cycle latency is an RTL change, which physical design can’t make on its own.

Block budgets turn into timing budgets and pin plans for hierarchical implementation. Accelerator datapaths with regular structure (MAC arrays, banked SRAM) floorplan well when the architecture sizes them to tile cleanly, as the TPU’s buffer-to-array pitch match shows.⁹

• Uncalibrated models. Simulator results without a calibration point are hypotheses. Calibrate against RTL or silicon of a similar block, and track the error as blocks arrive.
• Averages hiding tails. Average utilization looks fine while queues explode in bursts. Size for the tail. In the simplest queueing model (M/M/1: random arrivals, random service times, one server), the mean time in system is $\text{the service time} \div (1 - \text{utilization})$, so at 90% utilization it is already 10× the service time.²¹
• Too little concurrency. Bandwidth targets that ignore Little’s law fail because the requesters can’t keep enough transactions in flight to cover the latency.²²
• Coherence and ordering found late. Mixing coherent and non-coherent agents without a clear contract for ordering and cache maintenance (who flushes what, and when) produces bugs that only appear under specific traffic.
• Optimizing a flat optimum. Near the best pipeline depth, performance in the model above changes by under a percent per stage, while latch count and power keep climbing with depth.¹⁹
• Chiplet cuts in the wrong place. Cutting through a high-bandwidth path forces wide die-to-die interfaces, with their energy per bit, PHY area and latency.¹⁵

Optimal pipeline depth

Hartstein and Puzak model the time to run $N_{\mathrm{I}}$ instructions as a function of six quantities: the total logic delay $t_{\mathrm{p}}$ of the processor, the latch overhead $t_{\mathrm{o}}$ added by each stage, the number of stages $p$, the number of hazards $N_{\mathrm{H}}$, the average superscalar degree $\alpha$ (instructions issued per cycle), and $\gamma$, the average fraction of the pipeline stalled by each hazard. More stages shorten the logic part of each cycle ($t_{\mathrm{p}} / p$) but add overhead per cycle and lengthen each stall. Differentiating with respect to $p$ and setting the result to zero gives:¹⁹

The formula shows the opposing pulls. A higher superscalar degree $\alpha$ shortens the optimum, and fewer hazards $N_{\mathrm{H}}$ lengthen it. A larger ratio of logic delay to latch overhead ($t_{\mathrm{p}} / t_{\mathrm{o}}$) leaves more room for pipelining. Performance-only studies of several machines put the optimum at 8–10 FO4 per stage, logic plus overhead.¹⁹

Hrishikesh et al. reached a similar answer by simulating a high-performance core in 100 nm technology: about 6 FO4 of useful logic plus about 2 FO4 of overhead for integer code, 6 FO4 in total for floating point, and insensitive to latch and clock-skew overhead.²⁰ Power changes the answer. With the metric $\mathrm{BIPS}^3/\mathrm{W}$ (billions of instructions per second, cubed, per watt), Hartstein and Puzak’s optimum moved to about 22.5 FO4, roughly 7 stages, and it depends strongly on how the number of latches grows with depth. More dynamic power shortens the optimum; clock gating (stopping the clock to idle registers) lengthens it.¹⁹ The simulation above uses the performance-only form.

Roofline

$\text{Attainable performance} = \min(\text{peak compute},\ \text{peak bandwidth} \times \text{operational intensity})$, with intensity measured in operations per byte moved to or from DRAM. On log-log axes the bandwidth term is a line of slope 1 and the compute term is flat. The ridge point’s x-coordinate is the minimum intensity needed for peak; if it sits far right, only kernels with very high intensity reach peak. Ceilings add lower roofs for missing optimizations: in-core ones (instruction-level parallelism, SIMD, a balanced mix of floating-point adds and multiplies) and memory ones (unit-stride access, memory affinity, software prefetching).¹⁶

For accelerators, draw one roofline per level of the hierarchy. Raising intensity through tiling or a larger scratchpad moves a kernel right; raising bandwidth lifts the slope. The TPU v1 analysis is the standard worked example.⁹

Amdahl, Gustafson, and multicore

Amdahl’s 1967 argument was a limit: if a fraction $s$ of the work must run serially, even an infinite number of processors speeds the job up by at most $1/s$.¹⁷ In formula form, with serial fraction $s$ and parallel fraction $p$ of the time on one processor ($s + p = 1$), speedup on $N$ processors $= 1 / (s + p/N)$. Gustafson observed that problem size usually grows with the machine. Measuring $s'$ and $p'$ on the parallel machine instead gives scaled speedup $= s' + p'N$, and he reported speedups of 1016–1021 on a 1024-processor machine for applications with a 0.4–0.8% serial fraction.¹⁷ Use Amdahl when the spec fixes the job (latency per frame or per query) and Gustafson when it fixes the time and lets the work grow (throughput, batch size).

Hill and Marty add a hardware cost model. A chip has room for $n$ base-core equivalents (BCEs), each the area of one simple core, and a core built from $r$ BCEs delivers $\mathrm{perf}(r) = \sqrt{r}$, so doubling single-thread performance costs four BCEs. Asymmetric chips, with one big core and many small ones, can beat symmetric ones by a wide margin and are never worse: for $f = 0.975$ and $n = 256$ the best asymmetric speedup is 125.0 against 51.2 symmetric. Methods that look locally inefficient can be globally efficient because they shorten the serial phase while the other cores sit idle.¹⁸ The same reasoning justifies keeping a strong CPU next to a large accelerator. Multicore and vector units adds a power budget to the same models.

Cache misses: the 3Cs and AMAT

The 3Cs classify misses by simulation. Compulsory misses occur even in an infinite cache. Capacity misses occur even with full associativity and perfect replacement. Conflict misses are the remainder, caused by where blocks are allowed to sit. Larger capacity cuts capacity misses, higher associativity cuts conflict misses, and larger lines cut compulsory misses when nearby data is used together, each at a cost in hit or miss latency.¹⁰

AMAT nests: $\mathrm{AMAT} = t_{\mathrm{L1}} + m_{\mathrm{L1}} \cdot \bigl(t_{\mathrm{L2}} + m_{\mathrm{L2,local}} \cdot (t_{\mathrm{L3}} + \cdots)\bigr)$, where $t$ is a level’s hit time and $m$ its miss rate (local: counted over the requests that reach that level). It is a latency model. Out-of-order cores overlap misses, so the effective stall time also depends on how many misses are in flight at once (memory-level parallelism), which Little’s law ties to bandwidth.

NoC topologies

Topology fixes hop count, bisection bandwidth, node degree and layout cost. A bus is a single shared channel and a natural broadcast medium. A crossbar connects $n$ inputs to $m$ outputs in one hop and sits inside every router. Among direct networks (a router at every node), a ring is a $k$-ary 1-cube; a mesh drops a torus’s wraparound links and loses its edge symmetry, which concentrates traffic on the center channels. Under uniform random traffic, average hop count runs ring > mesh > torus, and a mesh has half the bisection bandwidth of the equivalent torus.¹¹

A torus’s long wraparound wires are tamed by folding, which interleaves the nodes to equalize link lengths. Concentration attaches several cores to one router to save area and power, at the risk of bottlenecks during bursts. Packet latency is head latency, roughly proportional to hop count, plus serialization latency $L/b$ for a packet of length $L$ on a channel of bandwidth $b$.¹¹ Chipyard’s Constellation is one open-source generator for exploring these choices.¹⁴

Queueing intuition for contention

Model a congested link or memory controller as an M/M/1 queue: requests arrive at random at rate $\lambda$, a single server handles them at rate $\mu$, and service times are also random. Utilization $\rho = \lambda/\mu$ must stay below 1 for the queue to be stable, and the mean time in system is $1/(\mu - \lambda)$, which is the service time divided by $(1 - \rho)$.²¹ At 50% load the latency doubles; at 90% it is 10×; near saturation it grows without bound. Real on-chip traffic often comes in bursts that this model doesn’t capture, so treat its curve as a best case.

Little’s law, $L = \lambda W$, is the companion tool. It gives the occupancy (buffer entries, outstanding requests) needed to sustain a throughput at a given latency, and it holds whatever the arrival pattern or service order.²² Together they explain why architects plan interconnect and memory controllers for well under 100% average utilization, and why a NoC’s latency curve stays flat until it suddenly isn’t.