Power and cooling

You know a cluster is racks of servers joined by a network. This chapter treats it as a thermodynamic and electrical system: a fixed amount of silicon power, multiplied up by conversion losses on the way in and by cooling overhead on the way out. The numbers that matter are rack power density (kW per rack), the efficiency of each stage of the power conversion chain, the heat each cooling method can remove per rack, and facility-level metrics such as PUE.

The pressure comes from one direction. Accelerator power has gone from around 100 W to more than 1,000 W per chip¹⁴, and scale-up domains want as many of those chips as possible within copper reach, which pushes them into one rack. The result is liquid-cooled racks at 120–140 kW today, roadmaps above 500 kW per rack before 2030, and a shift from 48 V to ±400 V DC distribution to feed them.¹⁶¹⁵¹⁴

We work through the chain stage by stage with real efficiency data, compare air, rear-door, direct-to-chip and immersion cooling with the heat-transfer equations behind them, take PUE apart using the industry’s measurement guidelines and the ITUE/TUE extension, and end with facility constraints: grid interconnection, water, and the power swings of synchronous training.

Separate three numbers that are often blurred: the modal density of a fleet (what most racks draw), the provisioned density of a design (what the busbar, breakers and cooling are sized for), and the measured draw of a rack under a given workload.

• Fleet reality. Uptime’s 2025 survey puts the average modal density at almost 9 kW (7.5 kW excluding sites whose typical rack is 30 kW or more), up from 8.3 kW the year before. Over 80% of operators report no rack above 30 kW; about one in eight has racks in the 30–59 kW band; cabinets above 100 kW exist but are rare. HPC and AI training are concentrated in relatively few sites.⁴
• Design targets. Liquid-cooled AI racks are provisioned at 120–140 kW today¹⁶¹⁵, HPC compute racks have exceeded 125 kW⁸, and vendor and hyperscaler roadmaps run from 100 kW toward 1 MW.⁵¹⁴
• Measured draw. AI servers rarely sit at nameplate. LBNL’s 2024 analysis of training runs found average node power of about 78% of the manufacturer rating for computationally saturated workloads.² Inference fleets swing more with demand.

Two forces set the density. The first is the per-accelerator power trend: socket power keeps rising while allowable case temperature falls, so the required case-to-coolant thermal resistance keeps shrinking.⁷ The second is network topology: the incentive to cut accelerator-to-accelerator latency.⁵ A scale-up domain built on passive copper reaches only a few meters, so its accelerators must share a rack (or a pair of racks), and the domain size times the accelerator power becomes the rack power. A 72-accelerator domain at around 1 kW per package, plus CPUs, switches and conversion loss, lands above 100 kW per rack whether or not the facility is ready.

The physical side effects compound: racks above 100 kW weigh over 1,800 kg (4,000 lb), need deeper and wider cabinets to fit manifolds and power hardware (one guide recommends at least 750 × 1,200 mm and 48U), and require floor and lift ratings older buildings may not have.⁵⁷

A concrete chain, from the ITUE/TUE paper’s study of the Jaguar supercomputer: the site distributes 13.8 kV; building transformers step down to 480 V AC; switchboards feed each cabinet; a cabinet PSU converts 480 V AC to 52 V DC; an intermediate bus converter on each blade makes 12 V DC; point-of-load regulators produce 1.3 V for processors and 1.8 V for memory. The authors estimated the two board-level stages at 84% combined.¹⁰ Modern AI racks keep the shape but move the PSUs into shelves feeding a 48 V-class busbar, with OCP Open Rack v3 specifying a 46–52 V DC interface for IT gear.¹³

Stage by stage, with what sets each loss:

• Transformers. Core (no-load) loss is paid at any load; copper loss scales with $I^2$. U.S. minimums are specified at 35% load: 99.14% for a 500 kVA three-phase low-voltage dry-type unit built 2016–2029, rising to 99.31% after April 2029.¹² Many transformers run most efficiently at about 20–50% load, so sizing them to the real load matters.⁸
• UPS. Double conversion (AC→DC→AC continuously) improved from 85–90% in the 1990s to 95% or more; eco/bypass modes reach about 99%.⁸ Across facility types the spread is wide: LBNL modeled 77–85% for small sites and 90–99% for hyperscale and AI.² Redundancy is the hidden cost: N+1 pairs sit at 50% load or less, and one guide’s example gains about 1.2 points of efficiency from a 10-point higher load factor.⁸ Training clusters checkpoint, which makes full UPS coverage less critical; some rack designs carry battery backup units instead.⁵¹⁵
• PSU / power shelf. Power-factor correction then isolated DC/DC. 80 PLUS Titanium (230 V internal redundant) requires 90/94/96/91% at 10/20/50/100% load; on a 1,000 W supply the gap between 92% and 96% is 87 W versus 42 W of heat.¹¹ Research front ends report 99% peak for the power-factor-correction stage, and a 400-to-50 V isolated stage 98.5% peak (97.5–98% at full load).¹³
• Intermediate bus and point of load. 48 V to about 1 V at hundreds of amperes is an extreme conversion ratio, so designs split it into a fixed-ratio stage and a multiphase regulator. Loss is dominated by conduction: package resistance, board copper, vias and inductors, which is why regulators keep moving closer to the package.¹³ On-die power delivery continues the same problem at chip scale; see Power planning.

Two consequences of the series structure. The end-to-end efficiency is the product $\eta_{\mathrm{chain}} = \prod_i \eta_i$, so input power is $P_{\mathrm{in}} = P_{\mathrm{load}} / \eta_{\mathrm{chain}}$. And every watt lost downstream is carried, with its own losses, by every stage upstream: a regulator loss is also a small PSU, PDU, UPS and transformer loss, and then a cooling load.¹³

Distribution voltage is the other lever. At a fixed conductor resistance, loss is $I^2 R = (P/V)^2 R$. A 120 kW rack at about 50 V draws 2,400 A, more than one busbar rated for 1,400 A carries¹⁶; a 1 MW rack at 50 V would need 20,000 A. Google’s move from 12 V to 48 V in the rack and now to ±400 V DC is this arithmetic; its first step, an AC-to-DC “sidecar” power rack, is reported to improve end-to-end efficiency by about 3% and frees the IT rack for accelerators.¹⁴ Higher-voltage DC buses shift the hard problems to insulation, DC fault interruption and protection, whose ecosystems are still developing.¹³

Every cooling method is a chain of thermal resistances from junction to outdoor air. ASHRAE frames the requirement as a case-to-coolant thermal resistance, $\theta_{\mathrm{ca}} = (T_{\mathrm{case}} - T_{\mathrm{coolant}}) / P$: as socket power rises and allowable case temperature falls, the required $\theta_{\mathrm{ca}}$ drops, forcing either colder coolant or a better path.⁷ The capacity of each method is set by its weakest link.

Sources: airflow and fan figures from ASHRAE⁷, method descriptions from DOE’s guide⁸, loop structure from Schneider’s liquid-cooling architecture paper.⁶

Liquid loops. A direct-to-chip system has three loops: the technology cooling system (TCS) through the servers, the facility water system (FWS), and a condenser water system to the outdoors where there is a chiller. The CDU between TCS and FWS does temperature control, flow control, pressure control (some run the TCS below atmospheric pressure so a leak draws air in rather than spraying water), fluid treatment, and heat exchange with isolation. Rack-mounted CDUs cover roughly 20–40 kW (rejecting to room air) or 40–80 kW (to facility water); floor-mounted liquid-to-liquid units run from 300 kW to over 1 MW.⁶

Water temperature classes. ASHRAE names liquid-cooling classes by their maximum facility supply temperature: W17, W27, W32, W40, W45 and W+.⁷ LBNL modeled liquid-cooled AI facilities at W45.² Warm supply lets dry coolers replace chillers for most of the year, but ASHRAE warns that rising socket power and memory pre-heat in shared loops may push future equipment back toward W32 or W27, colder and less efficient.⁷

Residual heat. Cold plates capture the processors (and sometimes memory), not the PSUs, NICs, drives and switch ASICs. Whatever is not captured must be removed by air, so a “liquid” rack can still exceed the room’s air capacity; a rear door is a common complement.⁸⁷

Immersion specifics. One vendor’s single-phase immersion module has a CDU rated for up to 200 kW with warm water and 368 kW with chilled water.¹⁸ As component power rises, single-phase natural convection hits its limit and designs need forced flow or two-phase.⁷ Many two-phase fluids are fluorinated chemicals. 3M announced it would exit all PFAS manufacturing, including fluorinated fluids, by the end of 2025¹⁹, and new dielectric fluids are being sought that address PFAS and global-warming concerns.⁵

The measurement guidelines that PUE’s sponsors (The Green Grid, ASHRAE, Uptime Institute, DOE, EPA and others) agreed on are precise about boundaries. Total energy is everything that crosses the data-center boundary: the utility meters of a dedicated building, or the datacenter’s sub-metered share of a mixed-use one. IT energy is measured at one of three points: UPS output, PDU output, or IT equipment input. Annual energy is strongly recommended, and the guidelines caution that comparing different data centers first needs analysis of factors such as reliability level and climate.⁹ By construction PUE runs from 1.0 to infinity.⁸

Four structural limits follow from that definition:

1. Boundary shifting. Everything inside the IT boundary is “useful.” The ITUE/TUE paper’s thought experiment: three identical sites, one with building fans and server fans, one with only server fans, one with only building fans. Their PUEs rank b best, then a, then c, although nothing says which uses least energy.¹⁰ The guidelines list in-rack and in-chassis cooling as supporting infrastructure, yet the UPS- and PDU-output readings they allow may still include non-IT devices installed in IT racks, such as fans.⁹ Direct-to-chip racks with in-rack CDUs and immersion tanks are exactly this case.
2. Denominator effects. More efficient IT raises PUE unless the infrastructure turns down with it.¹⁰ Raising inlet temperature lowers chiller energy but speeds server fans; with fan power scaling as speed cubed, a 30% speed-up more than doubles fan power, which PUE books as IT.⁸
3. Model versus reality. LBNL notes its simulated PUEs assume systems are commissioned and run as designed, which “is rarely the case” (air handlers fighting each other over humidity, for example).²
4. Scope. PUE excludes water and IT efficiency⁴, gives no credit for heat reused outside the datacenter or for on-site renewable generation⁹, and says nothing about work per joule.

$\mathrm{ITUE} = (\text{total IT energy}) / (\text{compute-component energy})$, with fans, PSUs, voltage regulators and the baseboard controller as overhead; $\mathrm{TUE} = \mathrm{ITUE} \times \mathrm{PUE} = \text{total energy} / \text{compute energy}$. In the paper’s Jaguar case study, the switchboards fed the cabinets 5,259.56 kW, PSU output 4,209.90 kW and board-level conversion 84% efficient, so compute received about 3,536 kW: $\mathrm{ITUE} = 1.49$, and with the building’s PUE of 1.25, $\mathrm{TUE} = 1.86$. For every kilowatt reaching compute, another 0.49 kW went to cooling and conversion inside the cabinets, nearly twice the 0.25 kW of overhead per IT kilowatt that the building’s PUE reported.¹⁰ The trade-off: TUE needs component-level power telemetry, and a cleaner definition of what counts as “compute” (the authors count storage; status LEDs and management controllers are overhead).¹⁰

At campus scale the binding constraints are interconnection capacity, lead time, and load shape, more than PUE.

• Demand. U.S. datacenter use was 192 TWh in 2024 (4.7%); the 2030 reference case is 649 TWh (11.8%), bounded by 521–843 TWh under compounded uncertainty, and datacenters would account for about a third of U.S. load growth over 2024–2030.¹ Infrastructure (mostly cooling) fell from 36% of datacenter electricity in 2018 to 31% in 2024 as load shifted to low-PUE, liquid-cooled AI facilities.¹
• Interconnection. Latency-sensitive training is why developers procure hundreds of megawatts at one site rather than spreading load; queues of 3–5 years follow.⁵ The IEA puts about 20% of planned projects at risk of delay, with transformer and cable lead times doubled in three years.³
• Load dynamics. A training iteration alternates a compute phase near TDP with a communication phase near idle. Across a site the swing reaches tens to hundreds of MW, with energy concentrated around 0.2–3 Hz, overlapping frequencies at which grid components such as turbine-generator shafts and long transmission lines can resonate. Proposed fixes span software (inject filler work when power drops), GPU firmware (ramp-rate limits and a minimum power floor, for example 65% of TDP) and rack energy storage.²⁰ A power floor trades energy for grid stability, a cost no PUE figure shows.
• Water. LBNL estimated an average site WUE just over 0.36 L/kWh through 2023, and stressed the coupling: water-cooled chillers and evaporative economizers lower PUE but consume water, while air-cooled chillers use none but more energy.²

The Expert view exposes every chain efficiency, the cold-plate capture fraction for direct-to-chip, and the arithmetic. It also brackets PUE’s “IT” against TUE’s “compute” on the bar and reports ITUE and TUE, so you can see how much loss PUE files under useful work. Try:

• Hold the chips fixed and move between air (8% fan share) and immersion (no fans). Compare the change in PUE with the change in TUE.
• With direct-to-chip, lower the capture fraction until the residual air heat, not the liquid loop, fails the rack.
• Set the PUE below what the electrical losses alone allow and see what the model does.
• Halve $\Delta T$ and watch the flow double: $Q = \dot{m}\, c_{\mathrm{p}}\, \Delta T$.

Cooling limits per mode (35, 60, 250 and 200 kW, plus 35 kW of residual air for direct-to-chip) and fan shares (8%, 8%, 3%, 0%) are round illustrative values chosen to sit inside the published ranges discussed above, not ratings of any product. The flow uses $c_{\mathrm{p}} = 4.18\,\mathrm{kJ/(kg \cdot K)}$ for water and $1.21\,\mathrm{kJ/(m^3 \cdot K)}$ for air; annual energy assumes 8,766 hours at the chosen average load; the Beginner view converts it to homes at 10,800 kWh a year.²¹

PUE, U.S. facilities serving AI (2024, modeled)

≈ 1.145

Jaguar ITUE / TUE

1.49 / 1.86

80 PLUS Titanium at 50% load (230 V redundant)

96%

Server fan share, dense air-cooled

10–20%

Sources: LBNL¹, the ITUE/TUE paper¹⁰, 80 PLUS table¹¹, ASHRAE.⁷

The ITUE/TUE worked example

Two sites with the same building (1.98 MW of infrastructure) and the same 10,000 servers, differing only in server power hardware. Per-server watts from the paper’s platform model:¹⁰

The efficient site draws about 4.64 MW instead of 5.3 MW for the same output, yet its PUE is worse. Only TUE ranks them correctly.¹⁰

Stage efficiencies used as defaults in the simulation

Anchors: eCFR¹², DOE⁸, LBNL², 80 PLUS¹¹, and the ITUE/TUE paper.¹⁰ The PDU and VRM defaults are assumptions.

Engineering trade-offs

• Coolant $\Delta T$ vs flow vs chip margin. For fixed $Q$, flow scales as $1/\Delta T$ and pump power, at fixed geometry, roughly as flow cubed. A large $\Delta T$ saves pumping and suits heat reuse, but raises the outlet temperature that the last cold plate in a series loop sees. Series loops minimize flow; parallel loops avoid pre-heat; ASHRAE expects designs to mix both.⁷
• Where to put conversion. Pulling AC/DC conversion into a sidecar rack frees IT space and gains about 3% end to end¹⁴; moving regulators closer to the package cuts conduction loss but raises local heat flux next to the hottest part.¹³
• Measurement boundary vs incentive. Moving fans or pumps across the IT boundary changes PUE without changing energy; use TUE, or at least report temperatures and boundaries with PUE.¹⁰
• Grid stability vs energy. Power floors and filler workloads smooth training swings but burn energy doing nothing useful; rack batteries cost capital and space.²⁰

Failure modes

• Condensation. Water below the room’s dew point sweats on coils and pipes; rear doors must stay above it unless the supply tracks the measured dew point.¹⁷
• Undersized margin. Rear-door sizing guidance adds 5–10% extra heat removal for airflow uncertainty; racks that are unevenly populated or have unusual airflow fall off the published curves.¹⁷
• Fluid chemistry and materials. Galvanic corrosion between mixed metals, biological growth and fouling in cold-plate channels; immersion fluids that attack plastics or void warranties.⁵⁷
• Hidden UPS capacity loss. When server fan power rises from 2% to 10% of server power, the same fans eat about 8% of UPS capacity, because they ride on the same backed-up feed.⁷
• Double counting. When estimating savings from a better converter, add the avoided loss and its cooling share ($\Delta P_{\mathrm{loss}} (1 + 1/\mathrm{COP})$); multiplying by PUE as well counts the cooling overhead twice.¹³

1. The rack energy balance

The simulation, and most first-cut facility models, build power outward from the silicon. With $N_{\mathrm{a}}$ accelerators at $P_{\mathrm{a}}$, $N_{\mathrm{s}}$ servers with overhead $P_{\mathrm{o}}$ (CPUs, memory, NICs), fan share $f$ and stage efficiencies $\eta$:

The $\max()$ matters: $\mathrm{PUE} - 1$ must at least cover the electrical losses between the meter and the rack, so with a 99% transformer, 96% UPS and 99.5% PDU, no facility can honestly report PUE below about 1.06 before it spends a watt on cooling. The general form, $P_{\mathrm{in}} = P_{\mathrm{load}} / \prod_i \eta_i$, and its sensitivity $\partial P_{\mathrm{in}} / \partial \eta_j = -P_{\mathrm{load}} / (\eta_{\mathrm{chain}}\, \eta_j)$, show that in a purely series chain a point of efficiency is worth about the same at any stage; in real, branched distribution the stages carrying the most power give the most leverage.¹³

Worked default: 72 accelerators at 1,000 W and 18 servers at 1,200 W give 93.6 kW of silicon. A 90% VRM raises that to 104.0 kW, 3% fans to 107.1 kW, a 96% PSU to 111.6 kW of IT power. The PDU, UPS and transformer bring it to 118.0 kW at the meter, and PUE 1.15 to 128.3 kW, leaving 10.3 kW for cooling and lighting. $\mathrm{ITUE} = 111.6/93.6 = 1.19$; $\mathrm{TUE} = 128.3/93.6 = 1.37$. PUE suggests 87% of facility power is “useful”; 73% reaches silicon.

2. Heat transport: $Q = \dot{m}\, c_{\mathrm{p}}\, \Delta T$

For water, $c_{\mathrm{p}} \approx 4.18\,\mathrm{kJ/(kg \cdot K)}$ and density ≈ 1 kg/L; for air, the volumetric heat capacity is about $1.21\,\mathrm{kJ/(m^3 \cdot K)}$ at room conditions.²² Their ratio, about 3,500 per unit volume, is the “roughly 4,000×” in Google’s comparison.¹⁴ For a 120 kW rack:

• Water, $\Delta T = 10\,\mathrm{K}$: $\dot{m} = 120 / (4.18 \times 10) \approx 2.9\,\mathrm{kg/s}$ ≈ 172 L/min.
• Air, $\Delta T = 15\,\mathrm{K}$: $\dot{V} = 120 / (1.21 \times 15) \approx 6.6\,\mathrm{m^3/s}$ ≈ 14,000 cfm, more than seven best-case floor tiles at 1,900 cfm each.⁷

The same equation explains the 40–50 kW air rack at about 5,000 cfm: 45 kW at a 15 K rise needs 2.5 m³/s, about 5,250 cfm.⁷

3. Fan and pump affinity laws

For a given fan or pump and system, $\text{flow} \propto \text{speed}$, $\text{pressure} \propto \text{speed}^2$, $\text{power} \propto \text{speed}^3$. DOE’s example: a 30% fan speed increase when inlet air rises from 77 °F to 91 °F more than doubles fan power ($1.3^3 \approx 2.2$).⁸ Applied to liquid loops, halving $\Delta T$ doubles flow and, through the same pipes and cold plates, multiplies pump power by roughly eight. That is why $\Delta T$ is a first-class design variable, not a detail.

4. Thermal resistance budget

The cooling requirement for a socket is $\theta_{\mathrm{ca}} = (T_{\mathrm{case,max}} - T_{\mathrm{coolant}}) / P$.⁷ An illustrative case: a 1,000 W package with an 80 °C case limit cooled by 40 °C water needs $\theta_{\mathrm{ca}} \le 0.04\,\mathrm{K/W}$ across the thermal interface, cold-plate base, channels and the fluid’s convective film. Doubling power to 2,000 W at the same temperatures halves the allowed resistance to 0.02 K/W; keeping 0.04 K/W instead would need coolant at $80 - 2000 \times 0.04 = 0\,^{\circ}\mathrm{C}$. In practice the designer gets the budget back from better interfaces, larger cold-plate area, more flow, or colder (less efficient) water: ASHRAE’s warning about W45 sliding back toward W32 or W27.⁷ The thermal side of the package itself is covered in Packaging and chiplets, and where the watts come from in Speed and power.

5. PUE, ITUE and TUE

All three are computed on annual energy.¹⁰⁹ Because ITUE and PUE share $E_{\mathrm{IT}}$, moving a fan from the building into the server lowers PUE and raises ITUE by offsetting amounts; TUE moves only if total energy does. Neither metric contains work. For comparing chips and systems on work per watt and cost, see Comparing chips.

6. Incremental savings from a better stage

For a fixed delivered load, improving a stage reduces conversion loss by $\Delta P_{\mathrm{loss}} = P_{\mathrm{in,old}} - P_{\mathrm{in,new}}$. If that heat would have been removed by mechanical cooling with coefficient of performance COP, the facility saving is about $\Delta P_{\mathrm{loss}} (1 + \alpha/\mathrm{COP})$, with $\alpha$ the fraction of the avoided heat that the active cooling would have handled.¹³ The review’s example: a 1 MW load through $0.990 \times 0.985 \times 0.984$ (95.96%) improved to $0.993 \times 0.985 \times 0.984$ (96.25%) cuts input by about 3.2 kW; at COP 4 the cooling saving adds about 0.8 kW.¹³

7. Annual energy

Here $u$ is the average load as a fraction of the provisioned power. A 100 MW facility at $u = 0.8$ uses about 701 GWh a year, the consumption of roughly 65,000 average U.S. homes at 10,791 kWh each.²¹ Real $u$ depends on workload: LBNL found about 78% of rated power for computationally saturated training nodes.²

Every layer in these guides ends up as a term in the same budget: picojoules per operation in the transistor, watts per accelerator in the package, kilowatts per rack, megawatts per site. Software decides how much useful work each of those joules buys, and also how the load swings: the training-loop power oscillations above are a software-shaped problem with a grid-scale consequence. Compilers, kernels, runtimes and collective libraries are the subject of the Software guide, coming soon.