Board and server

A server is a set of budgets that have to close at the same time: CPU lanes and switch ports, link bandwidth on each path the data actually takes, power from the busbar through several conversion stages, airflow or coolant, and board area. An accelerator-dense 8U system illustrates the scale: 8 GPUs, 2 CPUs, 8 single-port 400 Gb/s ConnectX-7 adapters for the compute fabric plus 2 dual-port ones for storage and management, 2 M.2 boot drives and 8 U.2 NVMe drives, 2 TB of DRAM, 6 × 3.3 kW supplies in 4+2 redundancy and a 10.2 kW maximum input.¹

The chapter works through each budget in turn:

• the host–accelerator split and the PCIe hierarchy (root ports, switches, endpoints), lane budgets and oversubscription, and how per-lane rates evolved from 8b/10b to PAM4 flits;
• CXL’s three protocols and device types, and what memory expansion costs in latency;
• power delivery from a 48 V-class busbar through intermediate and point-of-load conversion to sub-1 V rails at hundreds of amps;
• module and rack form factors from the Open Compute Project, and out-of-band management with a BMC.

Accelerator-to-accelerator fabrics are only touched on here; they have their own chapter, Scale-up fabrics.

Division of labor

The host owns the PCIe hierarchy, the I/O memory map and the main DRAM; each accelerator is an endpoint in that hierarchy with its own HBM, its own DMA engines and usually a dedicated scale-up fabric. The host link carries input batches, checkpoints, kernel launches and synchronization, plus network traffic when NICs exchange data with accelerator memory. Gradient exchange inside the server normally rides the scale-up links instead. The OAM base specification makes the split explicit: one or two x16 host links per module, and up to seven module-to-module links of up to x16–x20 lanes each.²³

The input pipeline is a host-side budget

Mohan et al. profiled nine models on 8-GPU servers and separated fetch stalls (waiting on I/O) from prep stalls (waiting on CPU decoding and augmentation). Their headline numbers: 3–24 CPU cores per GPU needed for pre-processing, up to 65% of epoch time spent in it, and an OS page cache that thrashes under the random access pattern of training.¹² The implication for server design is that “balanced SKUs” matter: cores per accelerator, DRAM capacity for caching decoded data, and drive and NIC bandwidth per accelerator are all first-class parameters, not leftovers.

Host-link topologies

Three host-link patterns recur. In the flat pattern, each device takes a root port directly and lanes run out quickly. In the switched pattern, PCIe switches fan out lanes and keep accelerator, NIC and drive traffic local; vendor documentation for one 8-GPU OAM platform shows the modules reaching dual-socket hosts through optional PCIe switches and retimers.³ In the coherent pattern, a CPU and accelerator share a coherent link wider than PCIe, or the accelerator speaks CXL. The rest of this chapter quantifies the first two and introduces the third.

Hierarchy

PCIe is a tree. Root ports on the root complex originate links; a switch is an upstream port and several downstream ports, each a logical PCI-PCI bridge, joined by internal routing; endpoints sit at the leaves.⁸ Each lane is a dual-simplex pair of differential pairs, so a x16 link is 64 signal wires (an OAM host link’s pinout lists 16 TX and 16 RX pairs).²³ Links train up from the lowest common rate and width; a x16 PCIe 5.0 port interoperates with a x1 device of any earlier generation.⁵

Per-lane bandwidth by generation

Usable per-direction bandwidth per lane is the transfer rate times the coding efficiency, divided by 8. Through PCIe 5.0 the coding is NRZ with 8b/10b (gens 1–2) or 128b/130b (gens 3–5); PCI-SIG’s table gives x16 per-direction bandwidth after encoding of 32, 64, 126, 252 and 504 Gb/s for generations 1 to 5.⁵ PCIe 6.0 keeps the 16 GHz Nyquist frequency of 5.0 but sends 2 bits per symbol with PAM4. The penalty is a raw error rate far above the $10^{-12}$ of earlier generations, so 6.0 adds a light-weight forward error correction (FEC), designed for a first-bit-error rate of $10^{-6}$, in front of CRC and link-level retry.⁴ FEC needs fixed-size codewords, hence flit mode.

Computed from the rates and flit layout in the PCI-SIG sources.⁵⁴⁶ Neither column counts transaction headers: in 128b/130b mode each packet also carries framing, sequence number and link CRC, which flit mode removes. PCI-SIG’s analysis puts flit mode ahead of 128b/130b in packet efficiency for payloads up to 512 bytes, for up to about 3× the effective throughput of 5.0 on small transfers, falling to about 0.98 efficiency at 4 KB payloads.⁴ Headline figures such as “256 GB/s for x16” at 64 GT/s are raw and bidirectional.¹⁶

Lane budgets and switches

Write the budget per root complex as $\sum w_i \le L_{\mathrm{CPU}}$ over directly attached devices, and per switch as $\sum w_i \le L_{\mathrm{down}}$, with the switch’s uplink width $w_{\mathrm{up}}$ charged to its CPU. The oversubscription ratio is $\rho = \sum w_i / w_{\mathrm{up}}$. For host-bound streaming with all devices active, each device gets $\min(1, 1/\rho)$ of its link. Peer-to-peer traffic between devices under one switch does not touch the uplink; when access control allows, “the transaction can route entirely within the PCIe hierarchy and never reach the root port,” while forwarding between root ports is not defined by the specification and the Linux kernel blocks it by default.⁹ NVIDIA’s GPUDirect RDMA guidance ranks the paths the same way: PCIe switches only is best, a path through one CPU is worse, and a path that crosses the inter-socket link may be severely limited.¹⁰ Two practical consequences:

• Pair each accelerator with its NIC (and ideally its drives) under one switch, so RDMA and storage DMA stay local.
• Spread switches across both sockets, and keep each process on the socket that owns its devices, so host staging doesn’t cross the socket link.

-[0000:00]-+-01.0-[01-06]----00.0-[02-06]--+-00.0-[03]----00.0  Accelerator (x16)
           |                               +-01.0-[04]----00.0  NIC, 400G (x16)
           |                               +-02.0-[05]----00.0  NVMe SSD (x4)
           |                               \-03.0-[06]----00.0  NVMe SSD (x4)
           \-03.0-[07]----00.0  CXL memory expander (x16)

Signal reach

Loss budgets grew from 22 dB at 8 GT/s to 28 dB at 16 GT/s and 36 dB at 32 GT/s, yet at 32 GT/s low-loss board material still allows only about 5 inches of system-board trace.⁷ A retimer splits the channel into two segments, each with a full budget.⁷ PCIe 6.0 holds reach roughly at 5.0 levels: PAM4 keeps the Nyquist frequency, though choosing a $10^{-6}$ FBER rather than networking’s $10^{-4}$ costs about 2–4 inches; in return FEC latency stays under 2 ns.⁴

Protocols and device types

CXL multiplexes three protocols on the PCIe PHY. Link training starts at 2.5 GT/s as PCIe, then negotiates CXL through alternate protocol negotiation. CXL.io reuses PCIe semantics for discovery, configuration, address translation and DMA; CXL.cache lets a device coherently cache host memory; CXL.mem lets the host (and other devices) load and store to device-attached memory as cacheable “host-managed device memory.” Type 1 devices implement .io + .cache, Type 2 all three, Type 3 .io + .mem.¹⁵ CXL 1.x/2.0 use a 68-byte flit (2 B protocol ID, 64 B payload, 2 B CRC); CXL 3.0 adopts PCIe 6.0’s 256-byte flit at 64 GT/s with no added latency, plus a latency-optimized variant.¹⁵¹⁶

From Das Sharma, Blankenship and Berger.¹⁵

Latency budget

The CXL port costs about 21–25 ns round trip on each side, and with about 15 ns of flight time through retimers the designers estimate a 57 ns end-to-end adder for a memory access across a link; the specification’s pin-to-pin target is 80 ns for a CXL.mem access. Their table estimates about 170 ns load latency from a CPU to a directly attached Type 3 device and about 250 ns through one CXL switch, and describes the direct case as similar to a remote-socket DDR access.¹⁵ Pond, a memory-pooling design for a public cloud, estimates 70–90 ns over NUMA-local DRAM for pools of 8–16 sockets and more than 180 ns for rack-scale pools.¹⁷ Measured hardware spreads widely: on a 4th-generation Xeon host, three CXL devices showed 35%, about 2× and about 3× the load latency of remote-socket DDR5, and the authors also found that emulating CXL with remote-socket memory can misstate latency.¹⁸

Bandwidth and pins

A x16 CXL link at 32 GT/s carries 64 GB/s raw per direction (128 GB/s at 64 GT/s).¹⁵ Pond notes that a bidirectional x8 port at a 2:1 read:write mix matches one DDR5-4800 channel,¹⁷ and Sun et al. note a PCIe 5.0 x8 memory device uses about 3× fewer pins than a DDR5 channel.¹⁸ The pin argument is the strategic one: CPU packages are pin-limited, and serial lanes deliver more bandwidth per pin than a parallel DDR bus, at the cost of latency.

What it is used for

• Capacity tiering. Hot pages in local DDR, colder pages on a Type 3 expander, managed by the OS’s NUMA policies.
• Pooling. Memory that would be stranded on one host is lent to another without a reboot; Pond reports a 7% DRAM cost reduction with performance within 1–5% of NUMA-local allocations.¹⁷
• Coherent accelerators. Type 2 devices share data with the host without explicit copies. On today’s AI servers the bulk of accelerator data still moves by DMA over PCIe semantics, so this is the least deployed of the three.

Stages and their efficiencies

The chain multiplies: $P_{\mathrm{wall}} = P_{\mathrm{load}} / (\eta_{\mathrm{PoL}}\, \eta_{\mathrm{IBC}}\, \eta_{\mathrm{PSU}})$. Open Rack V3 rectifiers are specified at 3 kW, a fixed 51 V output (switchable to 48 V), peak efficiency above 97.5%, and at least N+1 redundancy per shelf.²⁰ The rack base specification requires IT gear to operate from 46–52 V (51 V nominal) or 52–56 V (54 V nominal), and rates the IT-gear input connector’s power path at 100 A continuous, about 5 kW per connector at 50 V.¹⁹ The PSU specification’s rationale for a narrow voltage range is explicit: it “widely enable[s] 4:1 fixed ratio converters and downstream conventional 12V PoL converters.”²⁰ A fixed-ratio converter only divides the voltage; holding the output steady is left to the point-of-load stage after it, which is why the bus voltage itself has to stay in a narrow range.

Distribution loss

For a conductor of resistance $R$ carrying power $P$ at voltage $V$, $\text{loss} = (P/V)^2 R$. At 6 kW, 12 V means 500 A; 50 V means 120 A, and 1/17 of the loss in the same copper. The OAM pinout shows the same scaling at module level: its 48 V rail is 16 pins rated 16 A at 44 V, about 700 W; its 12 V rail is 27 pins rated 27 A, about 300 W at the 11 V minimum (the spec’s 12 V limit is 350 W).²³

Point-of-load regulation

Modern processors draw “hundreds of amperes of current at very low voltage (i.e., ≤ 1 V).”²² A multiphase buck parallels $n$ phases interleaved at $360^\circ/n$ with shared input and output capacitors. Interleaving cancels ripple current, which shrinks the capacitor banks; spreading loss across phases eases thermals; and during a load step the phases act in parallel, cutting effective output inductance by $n$ so the controller can slew current faster. Controllers add and drop phases with load to keep efficiency high.²¹ The application note’s guideline of 30–40 A per phase²¹ gives $n \approx I / 35\,\mathrm{A}$: about 25 phases for an 875 A rail (700 W at 0.8 V), before transient margin.

Efficiency is load-dependent. TI’s five-phase 12 V→1.8 V example holds above 90% from 5 to 200 A.²¹ Direct 48 V→1 V conversion is harder; the virtual-intermediate-bus regulator of Chen et al. (a 2:1 switched-capacitor stage to 24 V, then series-capacitor buck modules) reached 95.2% peak and 84.4% at its 640 A full load (power stage only).²² The drop at full load is the $I^2 R$ term in switches, inductors and board copper. It matters doubly in AI servers, whose accelerators can run near full load for long stretches of training.

Failure modes

• Transient droop. An accelerator stepping from idle to full load demands hundreds of amps in microseconds. Too few phases or too little capacitance and the rail dips below the chip’s minimum voltage; the chip errors or throttles. On-chip, the same problem continues inside the die (see Power planning).
• Current limits upstream. Eight accelerators synchronized by the training loop start and stop together, so the whole server’s load swings in lockstep, which the PSUs and busbar must ride through. Open Rack V3 rectifiers (3 kW rated) specify a pulse-power envelope and over-power protection that trips above 3.45 kW for 10 s or 3.6 kW for 100 ms.²⁰
• Redundancy that isn’t. Usable capacity is $(\text{installed} - \text{redundant}) \times \text{rating}$. A system loaded beyond N supplies has lost its redundancy without anyone noticing until a supply fails. One 8-GPU system with 4+2 supplies continues at reduced performance if three PSUs lose power, and will only boot with at least three working.¹

Why accelerators left the slot

The PCIe card electromechanical form factor was the quick path to market, but the OAM specification lists its problems for multi-accelerator systems: “excessive signal insertion loss from ASIC to PCIe connectors and baseboard, inter-card cabling complexity reducing robustness and serviceability, and limits the supported inter-ASIC topologies.”²³ OAM v1.5 answers with two 688-pin mezzanine connectors (rated to 56 Gb/s NRZ or 112 Gb/s PAM4), a 102 × 165 mm module, a 44–59.5 V input for up to 700 W (or 11–13.2 V for up to 350 W), one or two x16 host links, and up to seven module-to-module links that may be split into sub-links.²³ The baseboard then fixes the scale-up topology (fully connected, hybrid cube mesh and others), which is why module and baseboard are specified together.

Open specifications that matter here

The common thread is modularity: each spec draws a boundary (rack, module, card, management module) so that the parts on either side can evolve on different schedules and come from different suppliers.

What the BMC owns

The BMC is an out-of-band management SoC on standby power with its own NIC (or a sideband into a host NIC). OpenBMC’s feature list is a good inventory: chassis power control and power-state management, fan control, sensors, LEDs, inventory, logging, host watchdog, serial-over-LAN, remote KVM, full IPMI 2.0 with DCMI, a Redfish service and firmware update.²⁶ On an accelerator server it also reads accelerator and VRM telemetry, enforces power caps, and is the first responder for thermal events.

Redfish

Redfish was introduced in 2015 as a “RESTful interface over HTTPS in JSON format based on OData v4” and “a secure, multi-node capable replacement for IPMI-over-LAN,” covering health, sensors, power supplies and thresholds, reboot and console access.²⁷ Resources are linked by URI. DMTF’s public mockup of a 1U server exposes a CPU power sensor like this:²⁸

{
  "@odata.type": "#Sensor.v1_8_1.Sensor",
  "Id": "CPU1Power",
  "ReadingType": "Power",
  "Reading": 90,
  "ReadingUnits": "W",
  "SensingInterval": "PT0.01S",
  "PhysicalContext": "CPU",
  "Thresholds": {
    "UpperCriticalUser": { "Reading": 115, "Activation": "Increasing", "DwellTime": "PT0.03S" },
    "UpperCautionUser": { "Reading": 82, "DwellTime": "PT1S" }
  },
  "@odata.id": "/redfish/v1/Chassis/1U/Sensors/CPU1Power"
}

DC-SCM: management as a module

The OCP Datacenter-ready Secure Control Module (DC-SCM) puts the BMC, a hardware root of trust (HWRoT) and supporting logic on a module separate from the host processor board, connected by a standard interface. The Project Argus implementation uses an ASPEED AST2600 BMC running OpenBMC, an AST1060 HWRoT for secure firmware authentication, recovery and update, and a CPLD for I/O bridging, at under 20 W; the same module is meant to work across many host boards and generations.²⁵ The benefit is that security and management firmware are qualified once and reused, while host boards turn over with each CPU generation.

The full model: lane budgets per root complex and per switch (illustrative 144-lane switches, x16 up and 128 lanes down), per-direction bandwidth from GT/s × coding efficiency (128b/130b, or 236/256 for flits), and a power chain $P_{\mathrm{wall}} = P_{\mathrm{load}} / (\eta_{\mathrm{VRM}} \cdot 0.975 \cdot 0.96)$ against N+1 or N+N supply capacity. Expert controls add PCIe 7.0, VRM efficiency, a CXL Type 3 expander and a data-loading readout: drives → (switch uplinks, when data bounces through host memory) → accelerator links. On “8 accel, switched” (one switch per CPU, 8:1), raise the demand per accelerator past about 16 GB/s until the uplinks starve, then enable direct DMA.

PCIe 6.0 flit: TLP bytes / total

236 / 256

PCIe 5.0 channel loss budget

36 dB

48 V→1 V, 640 A regulator: peak / full load

95.2% / 84.4%

CXL link adder (est.) / switch path

57 ns / ≈ 250 ns

Links

• x16 per-direction bandwidth after encoding: 126 / 252 / 504 Gb/s for PCIe 3.0 / 4.0 / 5.0 (2010, 2017, 2019).⁵ PCIe 6.0: 64 GT/s PAM4, 256 B flits with 236 B TLP, 6 B DLP, 8 B CRC, 6 B FEC; FEC latency under 2 ns; expected retry time about 100 ns.⁴ PCIe 7.0: 128 GT/s, released June 2025.⁶
• Channel loss budgets: 22 / 28 / 36 dB at 8 / 16 / 32 GT/s; about 5 in of low-loss trace at 32 GT/s.⁷

Data path

• GPUDirect Storage: without it, data takes an extra copy through a bounce buffer in CPU memory; on some systems a direct path through a PCIe switch or a NIC acting as one “offers at least twice the peak bandwidth as compared to taking a data path through the CPU.”¹¹
• Input pipelines: 3–24 CPU cores per GPU for pre-processing; up to 65% of epoch time in it.¹²

CXL

• Estimated: 57 ns link adder; ~170 ns direct-attached Type 3 load; ~250 ns through a switch.¹⁵ Pond’s estimate for 8–16-socket pools: +70–90 ns over NUMA-local.¹⁷
• Measured: 35%, ~2× and ~3× remote-socket DDR5 load latency across three devices.¹⁸

Topology trade-offs

• Flat vs switched. Flat attachment gives every device an unshared path to host memory but exhausts root ports and pushes all peer traffic through the root complex, where forwarding between root ports is not defined by the specification.⁹ Switches buy fan-out and local peer-to-peer at the cost of $\rho{:}1$ host-bound bandwidth, added latency per hop, switch power, and a single point of failure per group.
• Generation vs reach. Each doubling tightens the loss budget relative to frequency; at 32 GT/s about 5 inches of low-loss trace is the practical limit before a retimer.⁷ Retimers add latency, power and board area; PCIe 6.0 trades 2–4 inches of reach for an FEC that stays under 2 ns.⁴
• Coherence vs simplicity. CXL.cache and Type 2 devices avoid explicit copies but couple the device to the host’s coherence protocol; most accelerator traffic still uses DMA with explicit synchronization.

Power trade-offs

• Two-stage vs direct 48 V→PoL. A fixed-ratio bus converter plus 12 V PoL reuses a mature ecosystem;²⁰ direct conversion removes a stage but must handle a large step-down ratio and the high input voltage stress, which is why research designs split it into merged stages.²²
• Phase count. More phases mean lower per-phase loss and better transients but more board area next to the package, which competes with memory, decoupling and signal escape.²¹
• Redundancy vs utilization. N+N survives a feed loss but strands half the installed capacity; N+1 per shelf (Open Rack V3’s minimum) is cheaper but covers only a single rectifier failure.²⁰

Failure modes worth instrumenting

• Link width and speed below capability (check negotiated vs capable link status on every boot), and rising correctable-error counts that precede a link dropping to a lower rate.
• Rail droop and VRM thermal throttling during synchronized load steps; PSU load per unit above the N-supply share.
• Data stalls: accelerator idle time correlated with input-pipeline CPU saturation or storage queue depth.¹²

1. Link bandwidth model

Per-lane, per-direction usable bandwidth is $b = R\eta/8$ (GB/s), with $R$ in GT/s and $\eta$ the coding or flit efficiency: 0.8 for 8b/10b, 128/130 for 128b/130b, and 236/256 if you count TLP bytes in a PCIe 6.0 flit.⁵⁴ A link of width $w$ gives $B = wb$. Transaction-layer overhead reduces this further for small payloads, and differently in each mode: in 128b/130b mode every TLP carries its own framing, sequence number and link CRC, while flit mode amortizes them, which is why PCI-SIG reports flit mode ahead for payloads up to 512 B.⁴ The simulator uses the plain $R\eta/8$ and ignores headers, flow-control credits and retries.

2. Lane allocation and oversubscription

The simulator assigns devices round-robin to ports (CPU root complexes, or switches interleaved across sockets), in order accelerators → NICs → drives, and admits a device only if its width fits the port’s remaining lanes:

for each switch k: cpu[socket(k)].used += 16            # x16 uplink
for kind in [accelerator, NIC, NVMe]:
  for i in 0 .. count(kind)-1:
    p = ports[i mod |ports|]
    root = p.isSwitch ? p : cpu[p.socket]
    if root.used + width(kind) <= capacity(root): root.used += width(kind)
    else: mark device i unconnected
rho(switch) = switch.used / 16                          # downstream / upstream lanes
hostShare   = min(1, 1 / rho)

Real platforms add constraints the model leaves out, for example root ports that split only in fixed patterns (x16, 2 × x8, 4 × x4) and lanes reserved for boot or management devices. Check the platform’s own documentation before trusting a lane count.

3. Data-loading bottleneck

Model the path from local NVMe to accelerators as stages in series, each with a capacity, and take the minimum:

• drives: $C_{\mathrm{d}} = n_{\mathrm{d}} \cdot \min(4b, \text{media limit})$ (the simulator uses 14 GB/s as an illustrative media limit);
• switch uplinks, only when data bounces through host memory and devices sit behind switches: $C_{\mathrm{u}} = n_{\mathrm{sw}} \cdot 16b$ per direction, because each byte goes up one uplink and down another;
• accelerator links: $C_{\mathrm{a}} = n_{\mathrm{a}} \, w_{\mathrm{a}} \, b$.

$\text{Feed} = \min(C_{\mathrm{d}}, C_{\mathrm{u}}, C_{\mathrm{a}})$; each accelerator receives $\min(D, \text{feed}/n_{\mathrm{a}})$ for a demand $D$ and is idle a fraction $1 - \text{that}/D$. With a bounce buffer, host DRAM sees $2 \times \text{feed}$ of traffic (one write, one read). Direct DMA removes both the uplink stage and the host-memory traffic, which is the mechanism behind GPUDirect Storage’s reported “at least twice the peak bandwidth” on switch-based systems.¹¹ The model assumes drives and accelerators are spread evenly over switches, so direct traffic stays local; it ignores CPU decode cost, which Mohan et al. show is often the real limit.¹²

4. Power chain

The simulator uses illustrative values: $f_{\mathrm{fan}} = 0.06$, $P_{\mathrm{BMC}} = 20\,\mathrm{W}$ (the order of the DC-SCM design’s under-20 W budget²⁵), $\eta_{\mathrm{IBC}} = 0.975$, $\eta_{\mathrm{PSU}} = 0.96$ and a default $\eta_{\mathrm{VRM}}$ of 0.90. Treating $\eta_{\mathrm{VRM}}$ as constant is the largest simplification: real point-of-load efficiency peaks at partial load and falls at full load (95.2% vs 84.4% in Chen et al.).²² Supply capacity is $(n - r) \times 3.3\,\mathrm{kW}$ with $r = 1$ for N+1 and $r = \lceil n/2 \rceil$ for N+N.

Currents follow from $I = P/V$ at each stage: the readout reports the 54 V input current, the 12 V intermediate-bus current and one accelerator’s core current at 0.8 V. The phase-count estimate $n \approx I / I_{\mathrm{phase}}$ with $I_{\mathrm{phase}} \approx 30\text{–}40\,\mathrm{A}$²¹ is a sizing rule of thumb; transient response, thermal limits and current-sense accuracy set the real number.

5. CXL latency, as a sum

Das Sharma et al. build the CXL.mem load latency bottom-up: CPU-side load-to-use under 100 ns (including the DRAM access the device performs), plus a CXL port round trip of 21–25 ns at each end, plus ~15 ns of flight time with retimers, which gives the 57 ns adder and an estimate near 170 ns; a switch adds $2 \times (21\text{–}25) + 10 + 10 \approx 62\text{–}70\,\mathrm{ns}$.¹⁵ The model’s assumption is that controllers hit the port-latency targets; the measured spread across real devices (35% to ~3× remote DDR5)¹⁸ is the size of the error when they don’t.