Scale-out networking

The scale-out network is the NIC-attached fabric that joins scale-up domains (servers or racks; see Scale-up fabrics) into one cluster. In current AI deployments it is a dedicated back-end network, physically separate from the front-end network that carries storage, checkpointing and management traffic.¹⁴ Each accelerator gets its own NIC, typically 400 Gb/s, so an eight-GPU server injects 3.2 Tb/s.¹⁴¹⁵

What makes it different from a general-purpose data-center network is the traffic. Cloud workloads produce millions of small flows that statistically even out. Training produces a few long, line-rate, synchronized flows per NIC, and every collective waits for its slowest member.¹⁵ That shifts the design questions:

• Topology: folded Clos (fat tree) built from identical switch chips, with radix and tier count setting scale, hop count and optics; how much oversubscription to allow and where; and whether to wire by rail.
• Transport: RDMA verbs over InfiniBand, over Ethernet as RoCEv2, or over the Ultra Ethernet transport; lossless links (credits or PFC) versus lossy links with smarter recovery.
• Congestion control and load balancing: ECN- and delay-based rate control, and what to do when per-flow hashing meets a handful of elephant flows.

The chapter takes these in order, then sizes real fabrics in “By the numbers” and works the math in “Under the hood.” The mapping from parallelism strategy to traffic is in Why the network looks this way; the copper and optical links themselves are in Optics.

Folded Clos, counted

Clos’s 1953 construction has an ingress stage of $r$ switches with $n$ inputs each, $m$ middle switches, and an egress stage mirroring the ingress. Each ingress switch has one link to each middle switch. The worst case for a new connection is that $n - 1$ of the ingress switch’s middle links and $n - 1$ of the egress switch’s are busy, on disjoint middle switches; so $m = 2n - 1$ middle switches guarantee a free path (his 36-port example: $n = 6$, $m = 11$).¹ Data-center fabrics fold the network so the ingress and egress switches are the same leaves, and settle for $m = n$ (one uplink per downlink). That makes the fabric rearrangeably non-blocking: for any permutation some set of paths saturates every host link, but finding it is a routing problem, not a guarantee.³

For radix $k$ and a non-oversubscribed fabric:

The three-tier row is Al-Fares et al.’s construction: $k$ pods of $k/2$ edge and $k/2$ aggregation switches, $(k/2)^2$ cores, $k^3/4$ hosts; for $k = 48$, 27,648 hosts and 576 equal-cost paths between hosts in different pods.³ Every switch is the same part, which is the point: Leiserson’s fat tree needed fatter links near the root, and a folded Clos gets the same effect by multiplying identical thin ones.² Leiserson also proved the shape is no accident: for a given volume of hardware, a fat tree can simulate any other network built from the same hardware with at most a polylogarithmic slowdown.²

Radix is the master variable

Because reach goes as $k^2$ or $k^3$, a 2× radix increase buys 4× or 8× more endpoints at the same tier count, or removes a tier at the same size. Removing a tier saves two switch traversals on the worst path, 40% of the switches per endpoint ($3/k$ versus $5/k$) and half the switch-to-switch links. That is why the per-chip SerDes budget matters so much: a 51.2 Tb/s chip built from 512 × 100G SerDes can run them as 64 × 800G ports, or group them more narrowly as 128 × 400G.¹⁹ At 400G per GPU, the 128-port configuration reaches 8,192 GPUs in two tiers instead of 2,048. Alibaba’s HPN exploits the same high radix, running each tier-1 chip with 128 × 200G ports down toward GPUs, to put 1,024 GPUs under a single layer of rail switches and about 15,000 GPUs in a two-tier pod, a size that “typically” needs three tiers.¹⁵

Bisection bandwidth and where to oversubscribe

Bisection bandwidth is the minimum, over all balanced cuts, of the capacity crossing the cut. For a folded Clos with $N$ endpoints at rate $B$ and leaf ratio $d{:}u$ (downlinks to uplinks, all same speed), cuts between leaves are bounded by the leaf uplinks: $NB/2 \times \min(1, u/d)$. Al-Fares et al. define oversubscription as the ratio of worst-case achievable host bandwidth to this bisection; 5:1 means only 20% of host bandwidth is available for some patterns.³

Where you oversubscribe matters as much as how much. Production AI fabrics keep the lower tiers at or near 1:1 and push the ratio to the top, then make the job scheduler respect it. Meta’s “AI Zones” are non-blocking two-stage Clos networks, with an oversubscribed aggregation layer joining zones, and a scheduler that places jobs to minimize cross-zone traffic.¹⁴ Alibaba runs tier 1 at 1.067:1 (128 × 200G down, 60 × 400G up per switch) and the aggregation-to-core layer at 15:1, assigning only pipeline-parallel traffic across pods.¹⁵

Rails, PXN and rail-only

A rail is the set of GPUs with the same local rank across scale-up domains, and a rail-optimized network puts each rail under its own leaves.¹⁶ Two things change:

• Locality. A rail group of eight leaves with $d$ downlinks each serves $d$ servers, and all same-rank traffic among them is one hop. Without rails the same eight leaves serve $d$ servers too, but only $d/8$ per leaf, so the one-hop set is 8× smaller.¹⁵
• Cross-rail traffic. GPU 1 in server A talking to GPU 2 in server B either climbs to the spine, or first moves over NVLink (or another scale-up link) to GPU 2 in server A and then rides rail 2.¹⁵ NCCL calls the second option PXN; NVIDIA measured all-to-all running more than twice as fast with it.¹⁷

Taken to its limit, this argues for removing the spine entirely. Wang et al. found that LLM training traffic is sparse and concentrated within rails, and that a “rail-only” network (a separate Clos per rail, cross-rail traffic forwarded through the scale-up domain) matches a rail-optimized network’s performance for models without mixture-of-experts layers, at 38–77% lower network cost than a full-bisection any-to-any Clos. For mixture-of-experts models, whose all-to-all must be forwarded, they estimate an 8.2–11.2% completion-time overhead on that traffic.¹⁶

RDMA’s assumption

RDMA NICs implement the transport (segmentation, ordering, acknowledgment, retransmission) in hardware and DMA directly to and from registered buffers, bypassing the host stack.⁸ NIC resources are tight, so classic implementations keep little per-packet state: in-order delivery is expected and loss recovery is go-back-N, which is cheap only when loss is rare.⁸⁹ Everything else in this section follows from that assumption.

InfiniBand’s link layer

InfiniBand satisfies the assumption at the link: the receiver on each link advertises credits, and the sender never sends beyond them.⁸⁹ Since the link is lossless and delivers in order within a virtual lane, the transport can stay simple: it was not designed to recover efficiently from loss and uses go-back-N.¹⁰⁹ Routing is computed centrally by a subnet manager that programs paths into every switch and recomputes them on link changes.¹⁰¹¹ Current InfiniBand switches add adaptive routing and in-network aggregation and reduction (SHARP), so part of a collective’s arithmetic happens in the switches.²⁰

RoCEv2 and PFC, in detail

RoCEv2 carries the IB transport in Ethernet/IPv4/UDP. The destination port is 4791; the source port is random per queue pair, and switches hash the five-tuple, so one QP stays on one path and different QPs can take different paths.⁷ Losslessness comes from PFC: when an ingress queue passes XOFF, the switch sends a pause for that priority upstream; below XON it sends a zero-duration pause to resume.⁷ Two costs follow:

• Headroom. Packets already in flight when XOFF is sent must be absorbed, so each lossless priority on each port reserves buffer that scales with MTU, reaction time and above all cable length. With 9 or 12 MB shallow-buffer switches and links up to 300 m, Microsoft could afford two lossless classes even though PFC defines eight priorities.⁷
• Collateral damage. Pauses are per priority, not per flow, and propagate back toward sources: head-of-line blocking, victim flows, unfairness, congestion spreading.⁸ In a stress test on one of its test clusters Microsoft also hit a PFC deadlock caused by the interaction of flooding and pause propagation, and in production it saw NIC “pause storms” from a malfunctioning NIC.⁷

Losslessness is also never complete. In a Microsoft lab test, a switch set to drop 1 in 256 packets showed that the NIC’s go-back-0 recovery (restart the whole message on any loss) turned that into livelock: links at line rate, zero goodput, because a 4 MB message of 4,000 packets never finished. Moving to go-back-N fixed it.⁷

Do you need a lossless fabric at all?

Mittal et al. argued that PFC is an artifact of RoCE NIC design: with selective retransmission and a bounded in-flight window on the NIC (their IRN design, about 3–10% more NIC resources), RoCE without PFC outperformed RoCE with PFC by 6–83% in their scenarios.⁹ The Ultra Ethernet transport (UET) takes the same position. It is explicitly designed for a best-effort network, and the specification says PFC should not be used anywhere in such a network because it violates the latency assumptions of UET’s congestion control.¹² UET still defines optional link-level tools: credit-based flow control (CBFC) for classes that must be lossless, and link-layer retry (LLR), which replays frames lost to bit errors at the link rather than end to end.¹² Its comparison of CBFC with PFC is a useful summary of the trade:¹²

Congestion signals and DCQCN

Flow control (credits, PFC) bounds buffer occupancy hop by hop; congestion control bounds it end to end by adjusting injection. DCQCN, the best-known scheme for RoCEv2, has three parts.⁸

• Congestion point (switch): ECN marking on egress queue depth via RED, as in DCTCP ($K_{\mathrm{min}}$, $K_{\mathrm{max}}$, $P_{\mathrm{max}}$).
• Notification point (receiver NIC): on a marked packet, send a CNP immediately if none was sent in the last $N$ µs, then at most one per $N$ µs ($N = 50\,\mu\mathrm{s}$ in Microsoft’s deployment).
• Reaction point (sender NIC): on a CNP, $R_{\mathrm{T}} \leftarrow R_{\mathrm{C}}$, $R_{\mathrm{C}} \leftarrow R_{\mathrm{C}}(1 - \alpha/2)$, $\alpha \leftarrow (1 - g)\alpha + g$. Without CNPs for $K = 55\,\mu\mathrm{s}$, $\alpha \leftarrow (1 - g)\alpha$. Recovery is QCN-style: five rounds of fast recovery $R_{\mathrm{C}} \leftarrow (R_{\mathrm{T}} + R_{\mathrm{C}})/2$, then additive and hyper increase. Flows start at line rate; there is no slow start.

That is a lot of knobs, and they interact with PFC thresholds and with each other. At 200G, Meta found that tight ECN thresholds avoided PFC but cut collective throughput in corner cases, and that NIC settings which shaved about 3% off all-to-all completion time made PFC activity 2–3× worse. At 400G, performance degraded with default settings, partly from firmware changes. It now runs 400G RoCE with PFC only and controls congestion in the collective library: a receiver-driven admission scheme in which a sender transmits a chunk only after the receiver signals buffer space, with those clear-to-send packets given high priority in the switches.¹⁴ UET’s NSCC uses RTT and ECN together, and its receiver-credit algorithm (RCCC) explicitly targets incast; the specification expects RCCC to work best on non-blocking fat trees and to be complemented by NSCC where the fabric is oversubscribed.¹²

Load balancing with few, large flows

ECMP spreads flows by hashing; Al-Fares et al. noted that it does not account for flow bandwidth and that implementations were then limited to 8–16 ways.³ With $F$ flows hashed uniformly onto $P$ paths, the expected number of paths used is $P\left(1 - (1 - 1/P)^F\right)$; for $F = P = 8$ that is 5.25, so about 34% of capacity sits idle while other links carry two or three flows (worked in “Under the hood”). Hedera quantifies the loss at 60.8% for a $k = 48$ fat tree with one flow per host at a time, falling to 2.5% at 1,000 concurrent flows per host.⁵ Collective traffic sits at the bad end of that curve, and with synchronous collectives the job’s step time follows the most loaded link, not the average.

Production responses, roughly in order of how much they change:

• Overprovision. Meta doubled leaf uplink bandwidth (1:2 undersubscribed) as a stopgap against flow collisions, calling it expensive.¹⁴
• Pin paths. Meta’s path pinning by destination worked only with whole-rack job placement and no failures; fragmented placement degraded training by up to more than 30%.¹⁴
• Add entropy. QP scaling posts each message over several queue pairs, and enhanced ECMP hashes on the QP number as well; Meta used a scaling factor of 16 for LLM workloads.¹⁴
• Reduce hashing stages. Alibaba’s dual-plane design keeps each NIC port’s traffic in one plane, avoiding polarization at the aggregation layer, and its large tier-1 segments keep most traffic off the aggregation layer altogether.¹⁵
• Spray or adapt. Packet spraying removes collisions outright. Dixit et al. showed that in symmetric multi-rooted trees the parallel paths have similar queueing, so even TCP tolerates the reordering; asymmetry (for example a failed link) hurts it.⁶ Adaptive routing picks ports by load in the switch and is built into current InfiniBand switches such as NVIDIA’s Quantum-2.²⁰

Ultra Ethernet’s design

UET makes the endpoint responsible for path choice. Switches keep plain ECMP and are expected not to rewrite the entropy value; the sender’s congestion-management sublayer sets a per-packet entropy (carried in the UDP source port, or in a UET entropy header) and so chooses among paths, spreading a connection across all of them and moving off paths that report congestion.¹² UET does not mandate a load-balancing algorithm.¹² Supporting pieces:

• Delivery modes. Reliable unordered (RUD), reliable ordered (ROD), reliable unordered for idempotent operations, and unreliable unordered; an MPI library might use ordered delivery for headers and unordered for bulk payload.¹²
• Signals. ECN marked at dequeue rather than enqueue, so the mark reflects the queue the packet actually waited in.¹²
• Trimming. A switch that can’t buffer a packet may cut it to its header and forward that in a separate class. For a spraying transport, loss is hard to infer from out-of-order arrival, so the trimmed header gives the receiver a fast, explicit loss signal.¹²

The topology numbers are exact for the construction described in “Under the hood” ($\text{uplinks} = \mathrm{round}(k/(1 + r))$). The two bars are a leaf-uplink bound, $\min(1, (u/d)/f)$, where $f$ is the share of a GPU’s traffic that must leave its leaf: uniform all-to-all (cross-rail hops ride the scale-up link, PXN-style), and one ring per GPU rank in placement order. They ignore congestion control, hashing and latency. The bottom panel hashes elephant flows onto equal-cost uplinks with a fixed hash; rehash to sample, and compare the slowest flow with packet spraying.

Per-switch capacity: Jupiter chip (2012) → 51.2T chip

0.64 → 51.2 Tb/s

HPN tier-1 ratio (128 × 200G : 60 × 400G)

1.067:1

DCQCN CNP interval / α timer

50 / 55 µs

ECMP loss: 1 vs 1,000 flows per host

60.8% vs 2.5%

Sizing, with real ratios

The pure-arithmetic sizes ($k^2/2$, $k^3/4$) move once real constraints apply. Alibaba’s tier-1 switch is a 51.2 Tb/s chip run with 128 active plus 8 spare 200G ports down and 60 × 400G up: 25.6 Tb/s against 24 Tb/s, 1.067:1. Each host has eight 2 × 200G NICs, one per GPU, with the two ports going to different switches (dual-ToR), so a host connects to 16 tier-1 switches; 16 switches collectively serve 1,024 GPUs in a segment. Alibaba reports that 96.3% of its LLM training jobs use fewer than 1K GPUs and so fit in one segment.¹⁵ A dual-plane tier 2 then reaches about 15K GPUs per pod in two tiers, with 15:1 oversubscription to the core.¹⁵

Meta’s numbers frame the job side: it has described RoCE clusters of up to 32,000 GPUs, trained a large Llama 3 variant on 16,000 GPUs of a 24,000-GPU cluster, and notes that because of multi-dimensional parallelism the largest single collective typically spans only hundreds of GPUs even in jobs of tens of thousands.¹⁴ That is the quantitative case for oversubscribing high in the tree.

Lossless Ethernet in practice

All figures from Guo et al.⁷

Tiers, radix and the optics bill

Per endpoint, a non-blocking two-tier fabric has $3/k$ switches and $k/2 \times k$ fabric links for $k^2/2$ endpoints, one per endpoint; a three-tier fabric has $5/k$ switches and two fabric links per endpoint. If every switch-to-switch link is optical, the transceiver count roughly doubles going from two tiers to three, on top of the extra hops. Meta’s leaf-to-spine links, for example, use single-mode fiber with 400G pluggable transceivers, while GPUs reach the leaf over copper within the rack.¹⁴ Radix is the lever that removes a tier, and lane speed and radix trade against each other on a fixed SerDes budget.¹⁹

Oversubscription is a scheduling contract

A u:d leaf caps cross-leaf throughput at u/d of line rate only for traffic that actually crosses; the delivered performance depends on what fraction of each GPU’s bytes must leave its block. Production designs make that fraction small by construction: Meta’s scheduler computes a minimum cut when splitting a job across AI Zones and assigns ranks accordingly;¹⁴ Alibaba places pipeline-parallel stages across pods so only that low-volume traffic meets the 15:1 core.¹⁵ The failure mode is a workload change, such as a mixture-of-experts model whose all-to-all crosses the top, landing on a fabric sized for rings.¹⁶

Rails versus generality

Rail-optimized fabrics keep a full Clos above the rails, so cross-rail traffic can still go up. Rail-only removes that layer: cheaper (38–77% below a full-bisection Clos in Wang et al.’s cost model), but cross-rail traffic must be forwarded through the scale-up domain, adding a “bandwidth tax” of extra bytes on those links and coupling network reachability to the scale-up fabric’s health.¹⁶

Lossless versus lossy, revisited

Lossless fabrics move the cost of congestion from retransmission into stalls: PFC’s headroom limits the number of lossless classes, and its pauses spread and occasionally deadlock.⁷ Lossy fabrics need endpoints with selective retransmission, bounded in-flight data and fast loss detection, which IRN showed costs a few percent of NIC resources⁹ and which UET builds in, with trimming to make loss explicit.¹² Credit-based flow control sits in between: lossless without PFC’s sensitivity to cable delay, but more complex and, per UET, worse at sharing buffer across ports.¹²

InfiniBand versus Ethernet

Technically the gap has narrowed from both sides. InfiniBand brings credits, in-order lossless links, central routing, adaptive routing and in-network reduction as one integrated design.¹⁰²⁰ Ethernet brings IP routing, multi-vendor switch silicon (in the two chips compared above, twice the radix at 400G) and shared operations with the rest of the datacenter, and is adding spraying, trimming, credits and link-level retry through Ultra Ethernet.¹² The choice in practice also turns on supply, staff experience and how much tuning an operator is prepared to own; Meta’s account of moving from DCQCN to receiver-driven admission shows how much of that tuning lands on the operator with RoCE.¹⁴

1. Counting a folded Clos with oversubscription

Let every switch have radix $k$, and let leaves have $u$ uplinks and $d = k - u$ downlinks (oversubscription $r = d/u$). The simulator sets $u = \mathrm{round}(k/(1 + r))$.

Two tiers. Each spine has one port per leaf, so $L = k$ leaves and $S = u$ spines. Endpoints $N = kd$; switches $k + u$; leaf–spine links $ku$; equal-cost leaf-to-leaf paths $u$; worst-case 3 switch hops. For $r = 1$ these reduce to $k^2/2$, $3k/2$ and $k/2$.

Three tiers. Generalizing Al-Fares et al.’s construction: $k$ pods, each with $k/2$ leaves and $u$ aggregation switches (each aggregation switch has $k/2$ ports down, one per leaf in its pod, and $k/2$ up); $\text{cores} = uk/2$, each with one port per pod. Endpoints $N = k \cdot (k/2) \cdot d$; switches $k \cdot k/2 + ku + uk/2$; pod-to-pod equal-cost paths $uk/2$; worst case 5 switch hops. For $r = 1$: $k^3/4$ endpoints, $5k^2/4$ switches, $(k/2)^2$ paths, matching the paper’s 27,648 hosts and 576 paths at $k = 48$.³

Rails. With eight NICs per server and no rails, a leaf holds $\lfloor d/8 \rfloor$ whole servers and $d \bmod 8$ ports go unused (in the simulator). With rails, groups of eight leaves hold $d$ servers, one NIC per server per leaf, so all $d$ ports are used and same-rank GPUs on $d$ servers are one hop apart.

2. Why $2n - 1$, and why fabrics settle for $n$

Take a three-stage Clos with $n$ inputs per ingress switch. A new call from a free input on ingress switch A to a free output on egress switch C fails only if every middle switch is already busy toward A or toward C. At most $n - 1$ middle links from A are busy (A’s other inputs) and at most $n - 1$ toward C, so with $2n - 1$ middle switches at least one is free on both sides: strictly non-blocking. Clos’s 36-line example uses $n = 6$ and $m = 11$.¹ With $m = n$ the network is only rearrangeably non-blocking: any permutation can be routed at full rate, but adding a connection may require moving others. Packet networks don’t hold circuits, so they take the cheaper $m = n$ and rely on routing (hashing, spraying, adaptive choice) to come close to the rearranged optimum.³

3. Bisection and the leaf-uplink bound

In a folded Clos with $d{:}u$ leaves, any traffic between leaves must use leaf uplinks, and each leaf has $uB$ of them for $dB$ of injection. For a balanced cut between groups of leaves, the crossing capacity is $NB/2 \times \min(1, u/d)$, which is the simulator’s bisection readout. More generally, if a fraction $f$ of each GPU’s traffic must leave its leaf, the sustainable injection rate per GPU is bounded by

That is the model behind the simulator’s two bars. It assumes perfect load balancing over the uplinks and enough non-blocking capacity above the leaves (true for this construction, where aggregation switches are 1:1).

4. Ring all-reduce and all-to-all, in bytes and in placement

For an all-reduce of $X$ bytes over $N$ participants, at least one participant must send at least $2(N - 1)/N \cdot X$ bytes, and a ring (reduce-scatter then all-gather around a ring) achieves that bound with each node talking only to its neighbors, contention-free on tree topologies.¹⁸ That neighbor-only property is what makes rings cheap on a fat tree:

• Ring, no rails. A leaf holds $s = \lfloor d/8 \rfloor$ servers. With one ring per GPU rank, visiting servers in placement order, each of the 8 rings has exactly one edge leaving the leaf: $f = 8/(8s) = 1/s$.
• Ring, rails. Leaf $i$ holds rank $i$ of $d$ servers; its single ring leaves once: $f = 1/d$.
• All-to-all. Of the $N - 8$ destinations outside a GPU’s own server, those under the same leaf are local. Without rails that is $8s - 8$; with rails, assuming cross-rail traffic first hops over the scale-up link to the destination’s rank (PXN), every GPU in the $d$ servers of the rail group is reachable at one hop, so $8d - 8$.¹⁷ $f = (N - \text{local} - 8)/(N - 8)$, close to 1 in any large network.

Worked example, $k = 64$, two tiers, 3:1 ($d = 48$, $u = 16$, $N = 3{,}072$): $u/d = 1/3$. Rings: $f = 1/6$ without rails, so throughput is $\min(1, (1/3)/(1/6)) = 100\%$. All-to-all: $f \approx 0.99$ without rails, so about 34% of line rate; with rails and PXN $f \approx 0.88$, about 38%. Oversubscription is nearly free for rings and nearly fully paid by all-to-all. At $k = 16$ and 3:1 ($d = 12$, $u = 4$), a leaf holds only one whole server without rails, so $f = 1$ and even the ring drops to a third; rails restore it to 100% by putting 12 servers per leaf group.

5. ECMP as balls into bins

Hash $F$ equal elephant flows independently and uniformly onto $P$ equal paths of capacity 1. A path carries nothing with probability $(1 - 1/P)^F$, so

and aggregate throughput, if each loaded link runs full, equals the number of paths used. For $F = P = 8$: 5.25 of 8, so 34% of capacity is idle. The maximum load grows roughly like $\ln P / \ln \ln P$ for $F = P$, so some link almost always carries two or three flows; those flows get 1/2 or 1/3 of line rate, and a synchronous collective finishes at their pace. With $F \gg P$ the relative imbalance shrinks, which is Hedera’s 60.8% versus 2.5% result: many small flows per host average out, one big one does not.⁵ Hashing at several tiers in a row with the same inputs makes it worse, because the choices are correlated rather than independent (polarization).¹⁵ Packet spraying turns the problem into $F \cdot (\text{packets})$ tiny balls, and balance becomes nearly perfect, at the price of reordering.⁶

6. DCQCN as a control loop

DCQCN is multiplicative decrease with a DCTCP-style estimate of the marking fraction: $\alpha$ tracks the recent probability that a CNP window saw a mark, and each CNP cuts the rate by $\alpha/2$.⁸ Its feedback is rate-limited to one CNP per flow per 50 µs, and its increase is timer- and byte-counter-driven rather than ACK-clocked, which is why its behavior under synchronized incast depends so much on parameters: in Meta’s experiments, ECN settings tight enough to avoid PFC cut collective throughput in some cases, and relaxed settings that trimmed completion time about 3% made PFC 2–3× worse.¹⁴ UET’s NSCC instead combines RTT and ECN at the sender, and RCCC hands out receiver credits, targeting the many-to-one case directly.¹²