Scale-up fabrics

A scale-up domain is the set of accelerators that share a memory-semantic fabric with roughly an order of magnitude more bandwidth per chip than the scale-out NIC. It exists because some forms of parallelism put communication on the critical path of every layer. Megatron-style tensor parallelism needs two all-reduces in the forward pass and two in the backward pass of each transformer layer,¹ and in practice it stays within the high-bandwidth domain, because across slower inter-server links its overhead becomes impractical.² The domain’s size therefore bounds the tensor- (and expert-) parallel degree; the mapping of parallelism onto the hierarchy is the subject of Why the network looks this way.

The design questions this chapter works through:

• Topology. How to divide a fixed SerDes budget: among direct neighbors (ring, full mesh, torus) or across parallel switch planes, and what each does to per-pair bandwidth, hop count and domain size.
• Semantics. Load/store/atomic access to peer memory, its ordering model, latency targets and link-level reliability.
• Collectives. The $\alpha\text{-}\beta$ cost model, the bandwidth lower bound for all-reduce, and which algorithm (ring, tree, halving-doubling, direct, dimension-wise, in-network) fits which topology and message size.
• Physical reach. Why current domains stop at one to a few racks of copper, and what open specifications target next.

Bandwidth ratios

The case for a separate fabric is mostly arithmetic. Published per-chip figures (all bidirectional) put scale-up an order of magnitude above the host link:

• Intel Gaudi 3: 1,200 GB/s of integrated Ethernet versus 128 GB/s of PCIe Gen5 x16, against 3.67 TB/s of HBM.³
• NVIDIA NVLink: 600 GB/s (12 links) for Ampere, 900 GB/s (18 links) for Hopper and 1,800 GB/s (18 links of 2 lanes at 200 Gb/s PAM4) for Blackwell.⁴
• Google TPU v4: six inter-chip links at 50 GB/s, against 1,200 GB/s of HBM.⁶

Even scale-up links remain several times slower than local HBM, which is why the parallelism layout tries to keep each chip’s working set local and exchange only activations or gradients.

Why not PCIe peer-to-peer?

PCIe does support device-to-device transfers, but in a host-rooted tree they cross shared switches and, for devices under different sockets, the CPU interconnect. Measurements on 8-GPU servers found PCIe peer bandwidth limited by switch sharing, and the bus a common bottleneck when GPUs communicate.⁵ A dedicated fabric adds bandwidth that scales with the number of accelerators rather than with host lanes, and lets the fabric protocol be tuned for small memory transactions and collectives.

Communication on the critical path

Tensor-parallel all-reduce volume per layer is proportional to batch × sequence × hidden size and does not shrink as the tensor-parallel degree grows, while the compute per chip does. So the communication fraction rises with the degree, and on slow links it dominates. That is the quantitative reason Megatron-LM keeps tensor parallelism within the server and uses pipeline parallelism across servers.² Collectives were already a major cost in classic HPC: a five-year study on a Cray T3E found over 40% of MPI time in allreduce and reduce.⁷

Dividing a lane budget

Treat each chip as having $P$ ports of bandwidth $b$ per direction (per-chip $B = Pb$). The topology decides how $P$ is spent:

The full mesh’s per-pair bandwidth falls as $1/(p-1)$, and its size is capped by port count: MI300X’s seven Infinity Fabric links (x16 at up to 32 Gb/s per lane) make exactly an eight-GPU mesh, with a quoted 896 GB/s peak peer-to-peer aggregate.⁸ A one-level switched domain is capped by switch radix instead: NVIDIA’s 72-port NVLink Switch chip sets the 72-GPU NVL72 domain, with each GPU’s 18 links going one to each of 18 switch chips.⁴ Broadcom’s Scale-Up Ethernet specification sketches the same plane structure with Ethernet switches: 64 accelerators, each with twelve 800G interfaces, through twelve 64-port switches, giving any pair up to 9.6 Tb/s.¹¹

Uniformity and routing

Point-to-point fabrics without forwarding expose topology to software. On the P100/V100 DGX-1 “hybrid cube mesh,” NVLink was not self-routed: traffic between GPUs without a direct link had to be relayed through an intermediate GPU chosen by the user, and measured latency rose from about 9 µs to roughly 2× (P100) or 3× (V100).⁵ The same study measured the switched DGX-2 as homogeneous: every pair reached 300 GB/s through NVSwitch, and the extra hop between baseboards cost almost nothing in latency.⁵ Irregularity also hurts collectives: when a scheduler hands a job an odd subset of an 8-GPU server, ring-based libraries leave some links unused, and packing spanning trees instead was up to 8× faster.¹²

Torus properties

A torus keeps degree constant, so it scales with short, cheap cables, but its bisection bandwidth grows only as the cut surface. Wraparound links matter: TPU v4 reports that they double both bisection bandwidth and the bandwidth of collectives such as all-reduce compared with a mesh, and it chose a 3D torus over its predecessor’s 2D torus partly for bisection.⁶ Collectives map well onto tori when they run dimension by dimension (see “Under the hood”); arbitrary all-to-all traffic does not, because it crosses many hops and shares links.

Semantics and ordering

A memory-semantic fabric carries read, write and atomic transactions with an ordering model, rather than NIC-mediated messages. UALink 1.0 defines exactly that, with the same ordering model for host-attached, local and remote accelerator memory, a 64-byte transaction-layer flit packed into 640-byte data-link flits, and address compression that reaches about 95% protocol efficiency for 256-byte writes (20 of 21 flits carry data in its example).¹³ Scale-Up Ethernet keeps the transport generic (XPU-specific command/response transactions used for put, get and atomics) and lists PGAS-style shared memory as the memory architecture, leaving registration and coherence services outside its scope.¹¹ The UALink white paper does not describe hardware cache coherence across the domain either. Collective libraries instead order their transfers with memory fences and flags; NCCL’s protocols differ mainly in which of the two they use.¹⁴

Little’s law sets the queue depth

$\text{Sustained bandwidth} = \text{outstanding bytes} \div \text{round-trip time}$. At $B = 800\,\mathrm{GB/s}$ and $W = 1\,\mu\mathrm{s}$, 800 KB must be in flight: 3,125 requests of 256 B. Halve the latency and the required depth halves. This is why scale-up specifications state latency targets in hundreds of nanoseconds to microseconds (UALink: under 1 µs request-to-response;¹³ SUE: under 2 µs round trip¹¹) and why they shave FEC latency, for example UALink’s reduced codeword interleaving, which trades burst-error correction for latency.¹³

Reliability and flow control

Memory traffic has no end-to-end retransmission in software, so reliability is pushed down a layer. NVLink applies CRC with replay on each link, ECC on datapaths and routing state, and a fabric manager that guards routing tables against out-of-range access.⁵ UALink combines Ethernet-PHY FEC, a 32-bit CRC per 640-byte flit, link-level retry and credit-based flow control, plus partitioning of a pod into isolated virtual pods.¹³ Lossless credit flow avoids drops but means congestion backs up hop by hop, so incast (many senders, one receiver) must be handled without head-of-line blocking, which SUE lists among its requirements.¹¹

Definitions and the decomposition

NCCL’s semantics: AllReduce leaves the element-wise reduction on every rank; ReduceScatter leaves $1/k$ of it per rank; AllGather concatenates $k$ contributions on every rank; AlltoAll sends chunk $j$ of rank $i$’s input to rank $j$.¹⁵ ReduceScatter + AllGather = AllReduce, and that decomposition is what makes all-reduce bandwidth-optimal.

The $\alpha\text{-}\beta$ model and the lower bound

Thakur, Rabenseifner and Gropp model a message as $\alpha + n\beta$ ($\beta = 1/B$ per byte), with $\gamma$ per byte for the local reduction, assuming full-duplex links and single-ported nodes.⁷ Patarasuk and Yuan prove that, without compression, a one-item all-reduce on $N$ processes needs at least $2(N - 1)$ partial results communicated, so some process must move at least $2(N - 1)/N$ of the data; a contention-free logical ring achieves it.¹⁶ The ring’s cost is then

(the paper’s form for its ring allreduce).⁷ Its bandwidth term is optimal; its latency term is the worst of the common algorithms. Equating the two $\alpha$ and $\beta$ terms (ignoring $\gamma$) gives a crossover at $n^* = p\alpha B$: below it the ring is latency-bound. With $p = 72$, $\alpha = 2\,\mu\mathrm{s}$ and $B = 800\,\mathrm{GB/s}$, $n^* \approx 115\,\mathrm{MB}$, which is larger than many tensor-parallel messages.

Choosing by message size

MPICH switches algorithms on size: recursive doubling ($\lg p$ steps, but $n \lg p$ bytes) for short all-reduces and Rabenseifner’s reduce-scatter + all-gather ($2 \lg p$ steps, $2(p-1)/p \cdot n$ bytes) for long ones.⁷ NCCL makes the same trade with both algorithms (ring and double binary tree) and protocols: its LL protocol has about 1 µs per hop but reaches only 25–50% of peak bandwidth, LL128 about 2 µs and roughly 95%, and Simple about 6 µs and near peak.¹⁴ Ring latency grows linearly with GPU count; double binary trees keep full bandwidth with logarithmic latency, which NVIDIA measured as up to 180× lower latency at 24,576 GPUs.¹⁷

Reporting: algbw and busbw

nccl-tests reports algorithm bandwidth $S/t$ and a “bus bandwidth” that multiplies it by $2(n-1)/n$ for AllReduce and $(n-1)/n$ for ReduceScatter and AllGather, so that a bandwidth-optimal implementation’s busbw approaches the per-rank link bandwidth regardless of rank count.¹⁸ Comparing busbw with the hardware’s per-chip bandwidth is the quickest sanity check of a collective’s efficiency.

Reach sets the domain

At 100–200 Gb/s per lane, electrical reach is short, and the specifications say so: UALink, built on IEEE 802.3 200G-per-lane electrical PHYs (KR/CR), targets cables under 4 m, 1–4 racks and up to 1,024 endpoints;¹³ Scale-Up Ethernet allows up to 10 m from accelerator to switch.¹¹ Past that, optics enter: TPU v3’s 2D-torus wraparound links between distant racks had to be optical, at over 10× the cost of electrical links, which pushed TPU v4 to electrically cabled $4^3$ blocks per rack (passive cables) with optical circuit switches only between blocks.⁶

Anatomy of a copper rack-scale domain

NVL72: 72 GPUs, each with 18 NVLink5 links (2 lanes at 200 Gb/s PAM4, 100 GB/s bidirectional per link), one link to each of 18 NVLink Switch chips of 72 ports; two switch chips per switch tray, nine trays.⁴ That is $72 \times 18 = 1{,}296$ GPU-side links, carried by four cartridges of more than 5,000 copper cables.¹⁹ The single switch level is set by radix: 72 ports per switch chip, 72 GPUs. The same slides show the domain growing from an 8-GPU hybrid cube mesh (2016) to an 8-GPU switched baseboard (2022) to 72 GPUs (2024).⁴

Torus designs reach rack scale differently. TPU v4 builds 4 × 4 × 4 blocks with passive electrical cables; each block exposes 16 links on each of its 6 faces (96 optical links) to 48 optical circuit switches, which assemble up to 64 blocks (4,096 chips) and can route around failed blocks.⁶ Trn2 uses a 4 × 4 2D torus inside a server at 1,024 GB/s per chip and rings between four servers at 256 GB/s per chip, a 4:1 bandwidth step at the server boundary.¹⁰

An $\alpha\text{-}\beta$ model over a 16-port chip ($B = 16b$ per direction). Pick the algorithm per topology: ring (two counter-rotating rings), binomial tree, halving-doubling (with whole-buffer fold-in for non-powers of two), direct one-shot on a full mesh, dimension-wise 2D rings on the torus, and, on the switched fabric, in-network reduction. The chart shows every algorithm on the current topology plus the endpoint bandwidth floor $2(p-1)/p \cdot n/B$; the readouts give busbw and the crossover size. Check that $n^* = p\alpha B$ for the ring, that a single ring on a full mesh wastes most links, and that in-switch reduction beats the floor.

Derived checks

• NVL72 aggregate: $72 \times 1.8\,\mathrm{TB/s} = 129.6\,\mathrm{TB/s}$, matching the quoted 130 TB/s all-to-all; the quoted 260 TB/s all-reduce is twice that, consistent with in-switch reduction halving per-GPU all-reduce traffic.⁴
• MI300X: x16 at 32 Gb/s = 64 GB/s per direction per link; $7\ \text{links} \times 64\,\mathrm{GB/s} \times 2\ \text{directions} = 896\,\mathrm{GB/s}$, the quoted peer-to-peer figure. One peer gets one link: 64 GB/s each way.⁸
• Gaudi 3: $8\ \text{cards} \times 21\ \text{ports} \times 200\,\mathrm{Gb/s} \times 2\ \text{directions} = 67.2\,\mathrm{Tb/s}$, the quoted node scale-up bandwidth; one peer gets $3 \times 200\,\mathrm{Gb/s} = 75\,\mathrm{GB/s}$ each way.³⁹
• TPU v4: $6 \times 50\,\mathrm{GB/s} = 300\,\mathrm{GB/s}$ of ICI against 1,200 GB/s of HBM, a 4:1 ratio.⁶

Topology trade-offs, quantified

• Mesh: no switch, shrinking pairs. Per-pair bandwidth $\approx B/(p-1)$. Collectives that use every link at once (direct reduce-scatter/all-gather) still reach near the $2(p-1)/p \cdot n/B$ floor, but point-to-point traffic between two chips (pipeline hand-offs, KV-cache moves) sees only one link: 64 GB/s per direction on MI300X.⁸
• Switched: uniform, but a hop and a radix. Any chip can push $B$ to any one peer, and in-network reduction becomes possible. Costs: switch silicon, trays, power and one switch traversal per transfer, and a one-level domain capped at the switch radix (72 ports → 72 GPUs).⁴
• Torus: cheap cables, $\sqrt{p}$ latency. Neighbor bandwidth $B/4$ (2D) or $B/6$ (3D); dimension-wise collectives recover full bandwidth, but all-to-all and irregular traffic share links over multiple hops. Wraparound links double bisection compared with a mesh but are the longest cables.⁶

Failure modes in practice

• Topology-blind allocation. Schedulers hand jobs fragments of a server (3, 5, 6 or 7 of 8 GPUs are common), and ring libraries then strand links; spanning-tree packing recovers up to 8×.¹²
• Relayed traffic. Without hardware routing, non-adjacent pairs relay through a peer at 2–3× latency.⁵
• Wrong regime. A bandwidth-tuned protocol on small messages pays ~6 µs per hop; a latency-tuned one on large messages wastes 50–75% of bandwidth (NCCL LL).¹⁴
• Incast and backpressure. Lossless, credit-based fabrics never drop, so congestion propagates; scale-up transports must spread traffic across planes and avoid head-of-line blocking.¹¹
• Blast radius. Larger domains raise the chance that some component fails during a job; reconfigurable optics let TPU v4 skip failed blocks and schedule around unavailable hosts.⁶

Ring all-reduce

Split the buffer into $p$ chunks. In reduce-scatter step $t$ ($t = 0, \dots, p-2$), rank $i$ sends chunk $(i - t) \bmod p$ to rank $i + 1$, which adds it to its own copy. After $p - 1$ steps rank $i$ holds the fully reduced chunk $(i + 1) \bmod p$. All-gather repeats the pattern without the addition. Each step moves $n/p$ bytes per rank on one link in each direction:⁷

The bandwidth term meets the Patarasuk–Yuan lower bound for algorithms in which only endpoints reduce.¹⁶ Running two rings in opposite directions on half the buffer each uses both directions of every link, which is how the simulator charges a physical ring.

Trees and recursive algorithms

• Binomial tree reduce + broadcast: $\lceil \lg p \rceil$ steps each way, but the whole buffer every step: $T = 2\lceil \lg p \rceil (\alpha + n\beta)$ (plus $\gamma$ on the reduce).⁷ Latency-optimal, bandwidth-poor.
• Recursive doubling: $\lg p \cdot \alpha + n \lg p \cdot \beta + n \lg p \cdot \gamma$, MPICH’s short-message allreduce.⁷
• Rabenseifner (halving-doubling): reduce-scatter by recursive vector halving and distance doubling, then all-gather by vector doubling and distance halving: $T = 2 \lg p \cdot \alpha + 2(p-1)/p \cdot n\beta + (p-1)/p \cdot n\gamma$. Optimal in both terms for powers of two, when the network carries the butterfly pattern without contention.⁷¹⁶ For other $p$, MPICH first folds the $r = p - p'$ extra ranks into neighbors with a half-buffer exchange and sends them the result at the end.⁷ (The simulator uses a simpler whole-buffer fold: 2 extra steps and $2n$ extra bytes.)
• Double binary trees: two complementary binary trees, arranged so that no rank is interior in both; pipelined, they approach full bandwidth with logarithmic latency. NCCL uses them alongside rings.¹⁴¹⁷ The simulator’s “Tree” is the unpipelined binomial version, to show the classic latency-for-bandwidth trade.

Direct (one-shot) on a full mesh

With $p - 1$ links per chip, reduce-scatter can be a single step: every rank sends chunk $j$ to rank $j$ over the dedicated link, and all $p - 1$ transfers proceed at once; all-gather mirrors it. With per-peer bandwidth $B_{\mathrm{pair}} = kb$ ($k$ ports per peer):

Since $k(p - 1) \approx P$, this is within rounding of the $2(p-1)/p \cdot n/B$ floor, with only two steps. A single ring on the same mesh uses 2 of the $p - 1$ links, so its bandwidth term is $(p-1)/2$ times worse, the mismatch the simulator’s expert view exposes, and the reason libraries pack many trees or rings onto point-to-point graphs.¹² (This formula is our derivation under the $\alpha\text{-}\beta$ model; real one-shot kernels also pay for $p - 1$ concurrent streams per chip.)

Dimension-wise algorithms on a torus

Mikami et al.’s 2D-torus all-reduce on an $X \times Y$ grid does reduce-scatter along rows, all-reduce along columns on the $1/X$-sized result, and all-gather along rows; they count $2(X - 1)$ GPU-to-GPU operations against $2(N - 1)$ for a single ring over all $N$ GPUs.²⁰ Ying et al. found TPU v3’s 1-D ring latency-limited at pod scale and ran two concurrent rings on halves of the payload, one along $X$ and one along $Y$, so both dimensions’ links stay busy, using wraparound links bidirectionally; they report twice the throughput of the 1-D algorithm.²¹ For an $R \times C$ torus with neighbor bandwidth $B/4$, a half that goes rows first costs

and the simulator takes the slower of the two halves. On 8 × 8 the bandwidth term equals a 64-chip ring’s (about $1.97\,n/B$) but the latency term is $28\alpha$ instead of $126\alpha$. A snake-shaped single ring on the same torus touches only 2 of 4 neighbor links and pays both the $126\alpha$ and twice the bandwidth term.

In-network reduction

With in-network reduction, each endpoint sends its $n$ bytes into the switch planes once and receives the reduced $n$ bytes once. With full-duplex links and pipelining, $T \approx 2\alpha + n/B$, below the endpoint floor $2(p-1)/p \cdot n/B$ by nearly 2×. NCCL’s NVLS algorithms use NVLink SHARP in NVSwitch for exactly this, and its CollNet algorithms use SHARP-capable network switches.¹⁴ The NVL72 switch chip lists 3.6 TFLOPS of in-network compute, and the system’s quoted all-reduce bandwidth (260 TB/s) is twice its all-to-all bandwidth (130 TB/s).⁴ The catch: arithmetic precision and format support are fixed by the switch, and the reduction needs buffer space and synchronization inside it.

Model assumptions

The $\alpha\text{-}\beta$ model ignores contention, assumes every rank sends and receives at once, treats latency as fixed per step, and charges nothing for chunking, protocol overhead or the reduction arithmetic ($\gamma$).⁷ Real libraries tune chunk sizes, channels and protocols, and the crossover sizes they choose are measured, not derived: MPICH, for example, set a 2 KB short/long boundary for reduce experimentally.⁷ Use the model to reason about scaling and regimes, and busbw from nccl-tests to measure the real thing.¹⁸