Why the network looks this way

You know a training cluster is thousands of accelerators joined by a network. This chapter is about why that network is built in two very different tiers. The reason is the parallelization strategy. A training job is a grid of parallelism degrees, data (DP), tensor (TP), pipeline (PP), expert (EP) and sequence or context, whose product is the accelerator count. Each dimension has a characteristic collective, volume per step and latency tolerance:

• TP issues several all-reduces per layer per microbatch, on the critical path.
• EP issues all-to-alls per MoE layer.
• PP sends point-to-point activations at stage boundaries.
• DP all-reduces (or reduce-scatters) gradients once per step, overlapped with the backward pass.

The published recipe, from Megatron-LM onward, is to keep TP inside a server-sized scale-up domain and let PP and DP span the scale-out fabric.⁴ Llama 3 ordered its dimensions [TP, CP, PP, DP] from innermost to outermost for exactly this reason.¹ We’ll derive the memory and traffic of each dimension with the standard open models (ZeRO, Megatron, GPipe, Korthikanti et al.), map them onto scale-up and scale-out links, and look at how real clusters and network designs follow from them. The collective libraries, compilers and kernels that implement all this belong to the Software guide; here they appear only as traffic.

DP of degree $d$ replicates the model and splits the global batch $B$. Gradients are synchronized once per step, after local accumulation over microbatches, with an all-reduce that a bandwidth-optimal ring performs by sending $2(d-1)/d$ of the gradient buffer per rank: a reduce-scatter followed by an all-gather.¹⁰¹¹ For $\Psi$ parameters in bf16 that is about $4\Psi$ bytes per rank per step, independent of $d$ for large $d$, which is why DP scales to thousands of replicas. Narayanan et al. note the ring time scales as $1 - 1/d$.⁴

Overlap is the second reason DP tolerates the scale-out network. DDP buckets gradients in reverse registration order and launches an all-reduce per bucket as soon as its last gradient lands, so most of the exchange hides behind the remaining backward computation.¹² The exposed part is roughly the last bucket plus whatever exceeds the backward time.

Memory is the limit. Mixed-precision Adam holds $16\Psi$ bytes of model state per replica.² ZeRO partitions it over the $d$ ranks:

Volumes here are in elements, following the paper; multiply by 2 for bf16 bytes. Stage 3’s extra $\Psi$ is the weight all-gather needed before both the forward and the backward pass.² FSDP issues those all-gathers per wrapped unit, prefetching the next unit’s gather during the current unit’s compute, and discards the gathered weights afterwards.¹³ With gradient accumulation it can either reduce every microbatch (sharded gradients, more traffic) or keep unsharded gradients locally (more memory).¹³ ZeRO stages 1 and 2 are therefore nearly free on the wire, while stage 3 pays in latency-sensitive gathers that a fast network or good prefetching must hide.

TP: column-then-row splits

For the MLP $Y = \mathrm{GeLU}(XA)$, $Z = YB$, Megatron splits $A$ by columns and $B$ by rows, so each of $t$ ranks computes $\mathrm{GeLU}(XA_i)\,B_i$ with no communication in between. The partial $Z_i$ are summed by one all-reduce (operator $g$) in the forward pass, and the conjugate operator $f$ all-reduces the input gradient in the backward pass. Attention splits $Q$, $K$, $V$ by heads the same way. That makes two all-reduces forward and two backward per layer.³ Each moves a $b \times s \times h$ tensor, so per layer per microbatch a rank sends $8bsh\,(t-1)/t$ elements, and the total grows with layers per stage × microbatches.⁴

Sequence parallelism replaces each all-reduce with a reduce-scatter and an all-gather along $s$ around the LayerNorm and dropout regions. Because a ring all-reduce is a reduce-scatter plus an all-gather, bandwidth is unchanged, and activation memory per layer drops from $sbh\,(10 + 24/t + 5as/(ht))$ to $sbh\,(34/t + 5as/(ht))$.⁵ TP also shrinks each GEMM by $t$, which lowers arithmetic intensity, and Narayanan et al. flag both this and the inter-server all-reduce as the reasons TP degrades beyond one server.⁴

PP: schedules and the bubble

With $p$ stages and $m$ microbatches per pipeline per step, GPipe runs all forwards, then all backwards. The bubble is $(p-1)(t_{\mathrm{f}} + t_{\mathrm{b}})$ against an ideal $m(t_{\mathrm{f}} + t_{\mathrm{b}})$, a fraction $(p-1)/m$.⁴ GPipe must stash activations (or stage inputs, with recomputation) for all $m$ microbatches. The 1F1B schedule (PipeDream-Flush) has the same bubble but at most $p$ microbatches in flight. Interleaving $v$ model chunks per device cuts the bubble to $(1/v)(p-1)/m$ at $v$ times the point-to-point traffic.⁴ Per microbatch, each stage boundary carries $bsh$ elements in each direction. With TP, the scatter/gather optimization sends $1/t$ of that tensor from each TP rank over its own NIC and re-gathers it over the scale-up fabric.⁴

EP: dispatch and combine

With $E$ experts and top-$k$ routing, an MoE layer’s FLOPs track $k$ while its parameters track $E$.⁸ Experts are sharded over an EP group usually carved from the DP dimension; Switch Transformer maps one expert per data-parallel core.⁸ Every MoE layer does a dispatch all-to-all and a combine all-to-all forward and the mirrored pair backward. Each sends about $k\,bsh\,(\mathrm{EP}-1)/\mathrm{EP}$ elements per rank, before capacity padding and imbalance. GShard measured all-to-all cost growing roughly as $\sqrt{D}$ for $D$ devices on its 2D TPU cluster; going from 128 to 2,048 experts raised its share of MoE-plus-Transformer time from 16% to 36%.⁷ Unlike ring collectives, all-to-all traffic crosses the bisection, so it is the dimension most sensitive to how the scale-out network is built.

Sequence and context parallelism

Megatron’s “sequence parallelism” (above) lives inside the TP group. Context parallelism (CP) instead partitions the whole sequence across ranks for every layer, including attention, and exchanges $K$ and $V$. Ring Attention passes K/V blocks around a ring and hides each transfer behind blockwise compute while compute takes longer.¹⁶ Llama 3 instead all-gathers $K$ and $V$, splitting the sequence into $2 \times \mathrm{CP}$ chunks for causal load balance. It reports the all-gather overhead as negligible, because attention compute grows as $s^2$ while the K/V exchange grows as $s$ (and grouped-query attention keeps $K$ and $V$ small).¹

Per rank per step, with $b = 1$, $m$ microbatches per pipeline, $L/p$ layers per stage and bf16, the standard volumes are as follows.⁴²⁵

Three observations follow.

• TP and EP scale with tokens; DP scales with parameters. TP traffic per step is proportional to $(\text{tokens per replica}) \times h \times (\text{layers per stage})$. DP traffic is proportional to the parameter shard and independent of batch size. Growing the global batch therefore grows TP and PP traffic, while DP traffic per token falls.
• Ring collectives need neighbor bandwidth; all-to-all needs bisection. A ring all-reduce of $n$ ranks finishes in about $(S/B) \cdot 2(n-1)/n$, where $B$ is the slowest link on the ring.¹⁰ An all-to-all pushes $(n-1)/n$ of every buffer across the cut between any two halves of the group.
• Overlap differs by dimension. DP all-reduces hide behind the backward pass with bucketing.¹² FSDP prefetches its gathers.¹³ PP sends and receives in both directions in parallel with compute. TP all-reduces separate dependent GEMMs and are largely exposed; Korthikanti et al. do overlap the extra all-gather that sequence parallelism adds in the backward pass.⁵ DeepSeek-V3 measured a computation-to-communication ratio of about 1:1 for its cross-node EP and designed DualPipe to overlap a forward chunk’s all-to-all with a backward chunk’s computation.⁹

Narayanan et al. measured the result at scale. Training their 1-trillion-parameter model on 3,072 GPUs, the effective bisection bandwidth used was 892 GB/s for pipeline point-to-point traffic and 12.9 TB/s for data-parallel all-reduces.⁴

Rank layout is the bridge between the parallelism grid and the topology. Llama 3 numbers ranks in the order [TP, CP, PP, DP], innermost first.¹ The simulator uses that order and slots an EP group, when present, just outside TP, so a rank’s index is $r = t_i + t\,(e_i + e\,(p_i + p\,d'_i))$, with $d' = d/e$. A dimension with stride $S$ and size $G$ lies wholly inside a domain of $U$ ranks when $SG \le U$, and every hop crosses domains when $S \ge U$. So:

• TP stays inside when $t \le U$.
• EP stays inside when $te \le U$.
• PP neighbors cross domains once $te \ge U$, which is almost always.
• DP, outermost, spans the whole job and crosses whenever the job is larger than one domain.

The simulator applies exactly this rule and times a group that spans domains at the scale-out rate. A ring is only as fast as its slowest hop, unless a hierarchical algorithm is used.

Hierarchical collectives change the picture for DP. The Rail-only paper analyzes a two-phase all-gather over $x$ GPUs per domain and $y$ domains. Across the network it moves $D(y-1)/x$ in total, all of it between same-rank GPUs (the same rail); inside each domain it moves $D(x-1)$.¹⁴ For GPT-1T on 4,096 GPUs in 256-GPU domains, a hierarchical all-reduce put 98% of the DP traffic on the scale-up fabric.¹⁴ Combined with TP kept inside domains, the scale-out traffic of a dense model is sparse and rail-local. That is why the authors argue spine switches are unnecessary, at 38–77% lower network cost and only 8.2–11.2% all-to-all overhead for MoE, by forwarding through the scale-up domain.¹⁴

MoE is where the scale-out network is stressed. DeepSeek-V3 used no TP at all, 16-way PP, 64-way EP spanning 8 nodes and ZeRO-1 DP on 2,048 H800 GPUs.⁹ To respect the 3.2× NVLink-to-InfiniBand gap (160 vs 50 GB/s), it limited each token to experts on at most 4 nodes. Each token crosses InfiniBand once per destination node, to the GPU with the same in-node index, and NVLink completes the fan-out, so a token can select an average of 3.2 experts per node without extra NVLink overhead.⁹ This is the same rail principle, applied to all-to-all by the model’s router rather than by the network.

Torus machines map dimensions onto physical axes instead of rank strides. TPU v4 places DP along one torus dimension and the two model-parallel factors on the others, and reports 1.2–2.3× gains from choosing topology and hyperparameters together.¹⁵

The Expert view adds activation recomputation, per-strategy collective types, time on the wire and an exposed-communication estimate. The model uses $12h^2$ parameters per layer, $b = 1$, bf16 with mixed-precision Adam (16 bytes per parameter before sharding), and Korthikanti activations with sequence parallelism. The first pipeline stage holds $\min(p, m) \times \lceil L/p \rceil$ layers of activations. Traffic follows the table above, and the bubble is $(p-1)/m$. Step time is $\text{compute} \times (1 + \text{bubble})$ plus exposed communication; DP hides behind up to two-thirds of compute. Bandwidths (450 and 50 GB/s) and 400 TFLOP/s of sustained math are illustrative. Try:

• Set the 406B model, 16,384 accelerators, TP 8, PP 16, ZeRO 3, a batch of 2,048 and an 8,192-token sequence, close to Llama 3’s Table 4. Compare selective and full recompute.
• Set the domain to 4 with TP 8 and read the exposed communication.
• With 64 experts, compare EP 8 (inside a domain of 8 when TP = 1) with EP 64.

Not modeled: interleaved schedules, embedding and logit layers, temporary buffers and fragmentation, MoE capacity padding and imbalance, hierarchical collectives, and latency. Treat the numbers as order-of-magnitude.

Points worth noting:

• Utilization holds up at scale. Narayanan et al. saw superlinear weak scaling from 1B to 1T parameters, 44% to 52% of peak, because larger models have larger GEMMs without proportionally more communication.⁴ Llama 3 lost two points of MFU going from 8K to 16K GPUs, which Meta attributes to the batch per DP group halving.¹
• PTD-P beat ZeRO-3 alone. For 175B and 530B models, Megatron’s combination outran ZeRO-3 without tensor parallelism by 70% when GPUs were doubled at constant batch, “due to less cross-node communication.”⁴
• Bandwidth ratios set the rules. Megatron-LM’s 2019 cluster had 300 GB/s between GPUs inside a server but 100 GB/s for the whole server to the network, shared by all its GPUs.³ DeepSeek-V3’s cluster had a 3.2× NVLink-to-InfiniBand gap.⁹ A smaller gap makes cross-node EP viable, while a larger one pins more dimensions inside the domain.
• Failures scale with $N$. Meta logged 466 job interruptions in 54 days of 16K-GPU training, 78% of the unexpected ones hardware-related, and still reached over 90% effective training time.¹

• TP degree vs GEMM efficiency. Every doubling of $t$ halves each rank’s GEMM width and doubles the relative cost of its all-reduces. Beyond $t = g$ (GPUs per server) the all-reduce also moves to slower links. Narayanan et al. measured up to 2× lower throughput for poor TP/PP splits even with fast inter-server links.⁴
• PP bubble vs memory vs batch. The bubble $(p-1)/m$ falls with $m$, but $m = B/(db)$, so at a fixed global batch (set by optimization, not hardware) more DP means fewer microbatches. Interleaving trades bubble for $v\times$ point-to-point traffic.⁴ Llama 3 had to modify its schedule because existing implementations required the per-GPU batch to be divisible by the number of stages.¹
• Microbatch size. Larger $b$ raises kernel arithmetic intensity but lowers $m$, which enlarges the bubble. The optimum is model- and configuration-dependent; Narayanan et al. found up to 15% throughput at stake.⁴
• Recomputation. Full recomputation costs one extra forward pass, up to 33% throughput at small batch. Yet it enabled batch sizes that gave up to 2× higher throughput through a smaller bubble.⁴ Selective recomputation with sequence parallelism cut activation memory 5× while removing over 90% of the recompute overhead.⁵
• ZeRO-3 vs model parallelism. FSDP avoids model-code changes but its gathers grow with re-gathering per microbatch and are latency-bound at small shard sizes. Llama 3 keeps it outermost and relies on prefetch; Megatron found PTD-P 70% faster than ZeRO-3 alone at 175B and 530B.¹⁴
• EP vs network topology. All-to-all is the one pattern that wants bisection bandwidth. Rail-only networks absorb it by forwarding through the scale-up domain at 8.2–11.2% overhead¹⁴, DeepSeek-V3 by node-limited routing⁹. The model’s routing and the network’s tiers end up co-designed.

The planner’s model, assembled step by step from the papers above. Notation: $\Psi$ parameters, $L$ layers, hidden size $h$, $a$ heads, sequence length $s$, microbatch size $b$, global batch $B$ (sequences), degrees $t$ (TP), $p$ (PP), $d$ (DP), $e$ (EP), and $m = B/(db)$ microbatches per pipeline per step.

1. Parameters and model-state memory

A GPT-style layer has about $4h^2$ attention weights ($Q$, $K$, $V$, output) and $8h^2$ MLP weights ($h \to 4h \to h$), so $\Psi \approx 12Lh^2$, ignoring embeddings.⁴ TP and PP divide a rank’s share to $\Psi/(tp)$. Mixed-precision Adam stores 2 + 2 + 12 bytes per parameter, and ZeRO stage $k$ divides the first $k$ of {optimizer, gradients, weights} by $d$:²

For MoE, expert weights are further divided by $e$ and sharded over the $d/e$ replicas of each expert. The sim uses top-2 routing over half-width experts ($4h^2$ each), so active parameters per token equal the dense model’s.

2. Activation memory

Korthikanti et al. count the bytes stored per layer in 16-bit training. Attention needs $11sbh + 5as^2b$, the MLP $19sbh$ and the two LayerNorms $4sbh$, for a total of⁵

With tensor and sequence parallelism every term divides by $t$. Selective recomputation drops the $5as/h$ term (the softmax and attention-dropout tensors that grow with $s^2$). Full recomputation keeps only $2sbh$ per layer, or $2sbh/t$ if sharded across the TP group.⁵ In 1F1B the first stage holds $p$ microbatches of $L/p$ layers, so⁵

This is independent of $p$. For GPT-3 ($a = 96$, $s = 2048$, $h = 12288$) the attention term is $5as/h = 80$, larger than 34, which is why selective recomputation pays off.⁵ The sim uses $\min(p, m) \cdot \lceil L/p \rceil$ layers, since a pipeline with fewer microbatches than stages never fills.

3. Ring collectives

A ring all-reduce over $n$ ranks of an $S$-element buffer runs $n-1$ reduce-scatter steps, in which each rank sends one $S/n$ chunk to its neighbor and adds the chunk it receives, then $n-1$ all-gather steps that circulate the reduced chunks.¹¹ Each element crosses $2(n-1)$ links in total, spread over $n$ links of bandwidth $B$, so¹⁰

This is why a ring is bandwidth-optimal and nearly independent of $n$, and also why it runs at the speed of its slowest link: put one scale-out hop in the ring and every step waits for it. Latency ($\approx 2(n-1)$ message times) dominates for small $S$, which is why TP’s many medium-sized all-reduces suffer most on a high-latency network.

4. TP and PP volumes

Each TP all-reduce operates on a $b \times s \times h$ activation tensor, and there are four per layer per microbatch (two forward via $g$, two backward via $f$), so the volume per rank, in elements per layer per microbatch, is³⁴

Per step, multiply by $L/p$ and $m$. With full recomputation the extra forward pass repeats its two all-reduces (×1.5). A PP stage boundary carries $bsh$ elements per microbatch per direction, divided by $t$ with scatter/gather.⁴ Worked example (the sim’s default): $s = 2048$, $h = 12288$, $t = 8$, $L/p = 12$, $m = 64$ gives $8 \times 2048 \times 12288 \times 7/8 \times 2\,\mathrm{bytes} \approx 352\,\mathrm{MB}$ per layer per microbatch, about 270 GB per step. PP moves $2 \times 64 \times 2048 \times 12288 \times 2/8\,\mathrm{bytes} \approx 0.8\,\mathrm{GB}$, and DP, with $2.72 \times 10^{9}$ local parameters and $d = 16$, moves $2 \times 5.4\,\mathrm{GB} \times 15/16 \approx 10\,\mathrm{GB}$.

5. The bubble

With flushes, the last stage starts $p-1$ forward slots late and the first stage ends $p-1$ backward slots early, so⁴

As a share of total step time it is $(p-1)/(m+p-1)$, GPipe’s $O\big((K-1)/(M+K-1)\big)$.⁶ Interleaving $v$ chunks gives $(p-1)/(vm)$. The sim reports $(p-1)/m$ in the Expert note and the share of step time in the status bar.

6. Compute time and exposed communication

A forward pass costs $24Bsh^2 + 4Bs^2h$ FLOPs per layer, and the backward pass twice that, so without recomputation a step is about $72BsLh^2\,(1 + s/(6h))$ FLOPs, or $96BsLh^2\,(1 + s/(6h) + V/(16Lh))$ with recomputation and the logit layer.⁴ The sim uses $6\,\Psi_{\mathrm{active}} B s + 12 L B s^2 h$ (×4/3 for full recompute) divided by $N$ accelerators at an assumed 400 TFLOP/s. It then estimates

Here each $t_X$ is volume ÷ bandwidth, at 450 GB/s if the group fits in a scale-up domain and 50 GB/s otherwise. The $\tfrac{2}{3}$ reflects that the backward pass, which DP traffic can hide behind, is two-thirds of the compute.⁴¹² This is deliberately crude: real TP overlaps partly, real EP can be fully hidden (DualPipe), and latency is ignored.⁹

7. A planning procedure

Narayanan et al.’s takeaways, turned into steps:⁴

1. Set $t$ to the smallest power of two up to the scale-up domain size that makes a layer’s state fit.
2. Set $p$ so that the model-parallel size $tp$ fits weights, optimizer state and activations in memory, with ZeRO-1 or -2 on top.
3. Let $d = N/(tp)$ and check that $m = B/(db) \gg p$; if not, interleave or reduce $p$.
4. Tune $b$ for kernel efficiency against the bubble, and choose recomputation to fit memory at the largest useful $b$.
5. For MoE, size $e$ so experts fit and all-to-all stays mostly inside the domain or node-limited, as DeepSeek-V3 did.⁹

Production systems search this space with measured cost models. Llama 3 notes it built a memory estimator and a performance-projection tool to explore configurations.¹