Optics

At the edge of every switch ASIC, NIC and accelerator sits a bank of SerDes lanes. A 51.2 Tb/s switch has 512 lanes at 100 Gb/s; its 102.4 Tb/s successor has 512 at 200 Gb/s.²⁴ How those lanes reach the other end of a link, over passive copper, retimed copper, a DSP-based pluggable module, a linear-drive pluggable, or optics co-packaged with the ASIC, sets the link’s reach, its energy per bit, and what happens when a part fails.

Two pressures drive the design space. First, electrical reach shrinks with every lane-rate doubling: OIF’s long-reach electrical projects went from 5 m of copper cable at 25 Gb/s to 3 m at 56, 2 m at 112 and 1 m at 224 Gb/s.³ Second, I/O power grows faster than the rest of the system. OIF’s 224G framework notes that SerDes power, on the host and in pluggable modules, is rising faster than any other part of system power.³ In OIF’s 12.8 Tb/s example the pluggable modules already drew nearly as much as the switch chip, 384 W against 432 W.²⁰ At 51.2 Tb/s, 64 DSP-based 800G modules at 15–16 W each add about 1,000 W of optics.¹⁶

This chapter works through the options in order of integration. It covers copper loss and SerDes equalization, what the DSP in a pluggable does and costs, what linear drive removes and what it couples together, and how CPO’s optical engines and external lasers change both the power budget and the failure domain. Energy figures are quoted per port end, in pJ/bit, with the host SerDes shown separately wherever the source allows.

The channel

A copper link is SerDes → package → host PCB → connector → cable → connector → host PCB → package → SerDes. Each segment’s insertion loss in dB is additive and, for transmission lines, linear in length, with a per-unit-length attenuation that has a conductor term growing as $\sqrt{f}$ once the skin depth is smaller than the conductor and a dielectric term growing as $f \tan\delta$.⁷ Losses are specified at the Nyquist frequency $f_{\mathrm{N}} = \mathrm{baud}/2$: 26.56 GHz for 100G PAM4 lanes at 53.125 GBd, 53.125 GHz for 200G lanes at 106.25 GBd.⁹¹⁰

Budgets by generation

OIF CEI long-reach projects, from the CEI-224G framework.³ The matching IEEE objectives are at least 2 m of twinax for 802.3ck’s 100 Gb/s lanes, alongside 28 dB at 26.56 GHz for backplanes,¹¹ and at least 1.0 m of twinax for P802.3dj’s 200 Gb/s lanes, alongside 40 dB die-to-die at 53.125 GHz for backplanes.¹²

The cable gets only part of the end-to-end budget. A baseline proposal for 802.3ck’s cable PHY divided a 29 dB channel at 26.56 GHz as 2 × (7 dB host + 1.6 dB connector) + 11.8 dB for 2 m of bulk cable and its wire terminations, against 12.62 dB for the 3 m bulk cable of the 50 Gb/s-lane generation, whose budget was quoted at 13.28 GHz.¹⁷ Roughly the same number of dB buys a cable two thirds as long when the Nyquist frequency doubles. At 200 Gb/s per lane the host side gets worse too: IEEE contributors flagged the host PCB and package loss budget as a challenge for 200 Gb/s-lane cables.⁸

What the SerDes pays for reach

Equalizing a 30–40 dB channel takes several stages: a transmit FFE (feed-forward equalizer, a short digital filter that pre-distorts each symbol), a receive CTLE (continuous-time linear equalizer, an analog boost of the high frequencies), and usually an ADC followed by a DSP running a longer FFE and a DFE (decision-feedback equalizer, which subtracts the echoes of symbols already decided), plus clock recovery. A survey of published 100 Gb/s-per-lane SerDes chips found 4.5–6.5 pJ/bit for long-reach designs handling 41.5–50 dB of loss and about 1.6–1.7 pJ/bit for extra-short-reach (XSR) designs handling 7–11.5 dB; the same survey projected 4–5 pJ/bit for 200 Gb/s-lane SerDes.¹⁸ One open analysis models a 224G long-reach SerDes at about 5 pJ/bit and a 112G short-reach (XSR) one at about 1 pJ/bit.¹⁹ That 3–5× gap between LR and XSR is the energy prize behind shortening the electrical path, which is what LPO partly and CPO fully do.

Two other costs come with the loss. PAM4’s four levels sit at a third of the NRZ eye height, so links run at raw bit-error ratios around $10^{-4}$ and rely on forward error correction to clean up. The Intel and TE 1 m, 36 dB demonstration ran at a pre-FEC BER of $3\text{--}4 \times 10^{-4}$.¹³ And retiming anywhere in the path, in an AEC or in an optical module, adds a full receive-and-transmit SerDes per lane at each point.

Active copper

IEEE’s terminology distinguishes a full active electrical cable, with a clock-and-data-recovery (retiming) device in each plug, from a non-retimed active copper cable whose plugs hold a linear redriver that equalizes the receive direction.¹⁴ Retiming splits the channel in three: host → plug, cable, plug → host, each with its own budget, so the cable segment can be much longer and thinner. Active copper cables at 100 Gb/s per lane were already on the market in 2023, at lower power than optical modules. At 200 Gb/s per lane, host PCB and package loss budgets squeeze passive cable, which is why fully retimed cables were proposed as the easier route.⁸ A 2024 Cisco comparison at 800 Gb/s put an AEC at about 8 W (about 10 pJ/bit) and 3 m or more, against under 1 W for a 1 m DAC and 16 W for a 20 m active optical cable.¹⁶

Architecture

A DSP-based (fully retimed) module presents a standard chip-to-module electrical interface to the host and a standard optical PMD to the fiber. The host side is a VSR-class channel: OIF’s CEI-112G-VSR covers about 16 dB at 29 GHz, against 28–30 dB for LR.⁹ Inside, the DSP retimes and equalizes the signal in both directions and converts between digital and analog; in the receive direction it retimes and cleans up the optical signal before sending it to the switch.²⁷ The analog front end is a modulator or laser driver and a TIA. Light comes from directly modulated lasers, electro-absorption modulated lasers or silicon-photonic modulators fed by continuous-wave lasers.

The key property is that the link is retimed twice. Electrical and optical budgets are closed independently against reference receivers, which is what makes multi-vendor interoperability tractable, and what LPO gives up.

Form factors and power classes

QSFP-DD and OSFP are both 8-lane form factors with defined power classes and thermal designs.²²²³ Power is the binding constraint on the faceplate: 32 or 64 cages, each dissipating up to about 30 W, must be cooled by front-to-back airflow. OIF’s 12.8 Tb/s example lists a typical 12 W per 400G port, 384 W for 32 ports beside a 432 W switch ASIC.²⁰

Where the module’s power goes

In NVIDIA’s description, a 1.6 Tb/s transceiver draws around 30 W, with the DSP consuming more than half, and the electrical path from switch ASIC to module is 14–16 inches of board trace or copper cable.²¹ One open analysis, from a photonics vendor, uses 16 pJ/bit for a DSP-based 112G-lane module on top of 5 pJ/bit for the host LR SerDes, 21 pJ/bit in total.¹⁹ Cisco’s 15–16 W per 800G pluggable comes to about 19–20 pJ/bit for the module alone.¹⁶ Treat the split inside the module as approximate: roughly half DSP, and the rest drivers, TIAs, lasers, their temperature control and the module’s microcontroller.

Failures

OIF puts current pluggable module failure rates at about 300 FIT, failures per billion device-hours.²⁰ With hundreds of thousands of modules in a cluster, that is a few failures a day, each a hot swap at the faceplate. Optical transceivers are among the most frequently replaced components in a datacenter.²⁸ A study of 350,000 optical links across 15 datacenters of a major cloud provider traced packet corruption to faulty and decaying transceivers, damaged fiber and dirty connectors, and found that the first repair attempt fixed a link only about half the time.²⁹

One budget instead of three

A DSP module closes three budgets: host TX → module RX (electrical, against a reference receiver), module TX → module RX (optical, against a reference receiver), module TX → host RX (electrical). In LPO the module’s driver and TIA are linear, so impairments accumulate from host TX through host channel, module driver, modulator, fiber, photodiode, TIA and far-end host channel to host RX, and only the far-end host receiver’s DSP removes them.³⁰ OIF’s co-packaging framework states the consequence for its linear-amplified interface: the drive signal must be relatively linear, without amplitude compression, and the ASIC SerDes “must compensate for the entire link, from SerDes Tx to SerDes Rx.”²⁰

That has three engineering consequences. Host SerDes must be LR-class or better, so LPO saves module power but not host power. The module’s analog front end must be linear over the full PAM4 swing, with controlled bandwidth and noise, which raises its own power somewhat. And the link’s performance is a property of a specific host-module-host combination, which is why the LPO MSA defines test points spanning host, module and fiber, plus an optional host-to-host link test, and why link diagnostics move into the host ASIC. The MSA’s 100G-DR-LPO specification defines 53.125 GBd PAM4 optical interfaces up to 500 m and host-module electrical interfaces for hosts with DSP-based SerDes and RS(544,514) FEC, building on OIF’s CEI-112G-LINEAR-PAM4.³¹ Since an LPO module does not retime, even a basic loss-of-lock indication means nothing in the module, and receive-side signal-quality monitoring (SNR, BER, FEC statistics) moves to the host ASIC.³⁰

Numbers

For a 51.2 Tb/s switch with 64 ports of 800G, Cisco’s per-port figures (under 9 W for LPO, 15–16 W for DSP modules) mean roughly 400 W or more saved, all of it in the optics.¹⁶ An open modeling paper uses 8 pJ/bit for a DR8 LPO module plus 5 pJ/bit of host LR SerDes, 13 pJ/bit in total against 21 for DSP pluggables, and expects LPO hosts to need DSP-based SerDes of 4.5–6 pJ/bit and co-design with the host platform.¹⁹ The half-retimed RTLR variant keeps the module’s transmit DSP, so the optical transmitter can still be tested against IEEE specifications, and makes only the receive path linear, using the host’s DSP SerDes.³³³²

Interfaces

OIF’s framework lists four host-to-engine interfaces: re-timed (the engine has a CDR/DSP, with the host using the CEI-112G-XSR extra-short-reach interface), linear amplified (no CDR/DSP in the engine; host SerDes compensates for the whole link), half-retimed, and direct drive, where the host drives the modulator directly and only the functions needed for a linear optical channel remain in the engine.²⁰ OIF’s 3.2 Tb/s CPO module agreement takes the XSR route: 32 × 112G XSR lanes, optics as 8 × 400G-DR4 or FR4, with an internal laser or an external one fed over polarization-maintaining fiber.³⁶ Energy-efficiency estimates in the framework, covering the engine-side electrical interface, CDR, photonics and laser but not the switch-side SerDes, are ≤ 15 pJ/bit for datacenter switching and HPC/AI, and 5–10 pJ/bit for AI training and disaggregation; a 3.2 Tb/s engine at 10 pJ/bit dissipates 32 W in roughly 20 × 20 mm (4 cm²), about 8 W/cm².²⁰

Engines and modulators

Engines differ mainly in modulator type and bandwidth density. Mach-Zehnder modulators are large, with optical bandwidth over 10 nm, and dominate the commercial market; micro-ring modulators are small and resonant, with optical bandwidth below 0.1 nm, which lets each ring add or drop its own wavelength for wavelength multiplexing.³⁵ Rings drift with temperature and fabrication variation, so they need closed-loop thermal tuning.³⁷ Ayar Labs reported a 1 nm resonance shift for a 16 °C change, and thermal transients of about 100 °C/s for an engine 3 mm from a 500 W ASIC, both of which its tuning loops must track.³⁸

Laser count and placement

Moving from externally modulated lasers in each pluggable to shared continuous-wave sources cuts laser count: NVIDIA cites 8 lasers per 1.6 Tb/s for a pluggable and 2 for its CPO engine, with micro-ring modulators and an external laser source module.¹ Each of its external laser source modules contains eight lasers and powers 32 transmit lanes; Spectrum-X Ethernet engines carry 3.2 Tb/s over 16 transmit and 16 receive lanes, Quantum-X engines 1.6 Tb/s over eight each way.³⁹ Broadcom’s 51.2 Tb/s design pairs its eight engines with field-replaceable remote laser modules.⁵

Accounting for the host SerDes

CPO figures usually exclude the switch’s own SerDes, and that matters. An open analysis, by a photonics vendor, charged 5 pJ/bit of LR SerDes to every option and got 21 pJ/bit for DSP pluggables, 13 for LPO and 12 for 2.5D CPO with its laser, concluding that CPO with large beachfront expansion, and so long-reach host SerDes, “does not deliver substantially different energy efficiency than linear pluggable optics.”¹⁹ The larger win needs the short electrical path to be exploited: XSR SerDes around 1.6 pJ/bit instead of LR at 4.5–6.5¹⁸, or direct drive. Intel’s Hot Chips 2024 slides make the same point, putting serial direct-drive CPO under 10 pJ/bit with 3–5 pJ/bit of that in the SerDes, and a die-to-die attached optical chiplet under 5 pJ/bit.⁴⁰

Twinax loss is $\alpha(f) = 1.1\sqrt{f} + 0.012 f\,\mathrm{dB/m}$ at $f_{\mathrm{N}}$; passive reach is a per-rate cable allowance divided by $\alpha$, calibrated to the 5/3/2/1 m reach classes, and a retimer at each end adds 24 dB of allowance. Energy bars split each port into host SerDes, DSP or retimer, driver/TIA or optical engine, and laser, using rounded values from the measured and modeled figures above; each card also shows the typical range. Try 200G at 1 m and at 3 m, and compare LPO with CPO, whose host SerDes is set at 3 pJ/bit, between XSR and LR.

Reliability arithmetic

Failure rates in FIT (failures per $10^{9}$ device-hours) add across the parts a system needs. OIF’s reliability targets for a 51.2 Tb/s CPO switch with sixteen 3.2 Tb/s engines are 10 FIT per engine (lasers excluded), 50 FIT per 3.2 Tb/s laser source, 100 FIT for the rest of the system, and a total annualized failure rate under 1%, against roughly 300 FIT for today’s pluggable modules.²⁰ Sixteen engines at a pluggable-like 300 FIT would be 4,800 FIT, an engine failure about every 24 years per switch; across 10,000 switches, more than one a day, each a board replacement. The framework adds that redundancy, and modes that run with fewer lanes, may be needed to reach system targets.²⁰

Failure domain and repair time

Availability is failure rate × impact × time to repair. A pluggable failure takes one port down for a minutes-scale swap. A CPO engine serves dozens of lanes: in Broadcom’s Bailly each of eight engines serves 16 of the 128 400G ports, so losing one takes out 16 ports at once.⁵ AI training is less tolerant of link loss than traditional datacenter networks, which rely on redundant links; OIF notes that AI applications are typically less tolerant of a link failure because of their higher aggregate bandwidth and connectivity requirements.²⁰ The design answers are external, field-replaceable lasers, which the ELSFP agreement exists to provide;³⁴ burn-in and known-good-engine screening; laser sparing and power boosting; and graceful lane degradation.

Field data

Meta’s reported results so far cover about 15 million CPO device-hours and two million pluggable-module hours, with no unserviceable CPO failures, about 5× better MTBF for CPO and about 65% lower optics power.⁴⁸⁴⁹ In about one million 400G-equivalent port device-hours of Meta’s high-temperature lab testing, no link flaps were observed.⁴⁹ This is early data, from one operator and one product generation.

Other costs

• Supply chain and test. CPO turns optics from a separately purchased, separately qualified commodity into part of the switch build. A package with many engines multiplies integration yield risk, so engines must be screened for defects before attach.²⁰
• Fiber management. Besides its data fibers to the faceplate, an engine with an external laser needs polarization-maintaining fibers from the laser module, all routed inside the box.³⁶
• Power density. About 8 W/cm² for a 32 W, 20 × 20 mm engine, far above what optical components usually face.²⁰
• Host SerDes. LPO requires LR-class host SerDes, and CPO’s energy win is largest only when the host interface shrinks to XSR or direct drive.¹⁹

1. A copper reach model

Model twinax loss per meter at the Nyquist frequency as a skin-effect term plus a dielectric term:

The conductor term follows from skin depth shrinking as $1/\sqrt{f}$; the dielectric term from loss tangent absorption, proportional to $f$.⁷ Loss in dB is linear in length because attenuation is exponential in distance. If the SerDes can equalize a total budget $B$ and the host side (packages, traces, vias, connectors) takes $H(f)$, the cable gets $A = B - H(f)$ and

Worked example from the 802.3ck cable baseline proposal: $B = 29\,\mathrm{dB}$ at 26.56 GHz, $H = 2 \times (7 + 1.6) = 17.2\,\mathrm{dB}$, so $A = 11.8\,\mathrm{dB}$ for a 2 m cable, about 5.9 dB/m.¹⁷ If loss were pure skin effect, doubling to 53.1 GHz would raise $\alpha$ by $\sqrt{2}$ to about 8.3 dB/m; with the dielectric term it is a little more. Holding $A$ constant would give about 1.4 m. In practice $H$ grows too at 200 Gb/s, which is why the P802.3dj objective is 1.0 m.¹²⁸ The sim uses exactly this model with per-rate allowances fitted to the standards’ reach classes.

2. Why reach falls faster than the loss curve

PAM4 puts three eyes in the swing that NRZ uses for one, so each eye is a third the height: a $20 \log_{10} 3 \approx 9.5\,\mathrm{dB}$ SNR penalty at equal swing and noise. Equalization recovers loss by boosting high frequencies, but it boosts noise and crosstalk with them, so the usable budget $B$ is bounded by the receiver’s noise floor, not by how many equalizer taps can be built. That is why budgets have stayed in the 25–40 dB range across generations while the frequencies they apply at have quadrupled.³¹² Links therefore run at pre-FEC error ratios around $10^{-4}$ and lean on FEC.¹³

3. Energy accounting

Energy per bit is power over throughput, and the unit conversion is convenient:

So a 51.2 Tb/s switch’s ports cost 51.2 W per pJ/bit. Charging each option per port end (one side of the link, which is what a switch pays), with illustrative values from the sources above:

Where the numbers come from: host LR SerDes 4.5–6.5 pJ/bit and XSR about 1.6¹⁸; AEC about 8 W per 800G end, or 10 pJ/bit, which the table rounds up to 11¹⁶; DSP module 16–20 pJ/bit with the DSP over half¹⁹¹⁶²¹; LPO module 8 to under 11 pJ/bit; CPO engines and lasers about 7 pJ/bit together, under 7.5 in Cisco’s comparison, split here as 4.5 and 2.5.¹⁹¹⁶ The CPO host SerDes value of 3 is a midpoint between XSR and LR; Intel quotes 3–5 pJ/bit for SerDes driving CPO directly.⁴⁰ The splits within the optics (driver versus laser, DSP versus the rest) are approximate.

4. Laser power budget for an external source

In an external-laser design the laser’s light passes a connector at the laser module, fiber, a connector at the engine, fiber-to-chip coupling, splitters and the modulator before it carries any data, and each loss is paid in laser power. In dB terms:

Electrical power is then $P_{\mathrm{laser}}$ divided by the laser’s wall-plug efficiency, plus its cooling. OIF notes that external sources improve reliability and thermal environment “at the cost of higher insertion loss,” compensated by increasing laser output power, and that integrated lasers have fewer losses to overcome.²⁰ Sharing one high-power laser across several lanes (the 1:N splitting cases in OIF’s framework)²⁰ cuts laser count, at the price of a larger loss per lane and a larger blast radius if that laser fails.

5. Reliability: from FIT to outages

For constant failure rates, $\lambda_{\mathrm{system}} = \sum_i N_i \lambda_i$ and the annualized failure rate is $\mathrm{AFR} \approx \lambda \times 8760\,\mathrm{h}$. OIF’s targets for a 51.2 Tb/s CPO switch:²⁰

With lasers field-replaceable, only the engine and rest-of-system terms (260 FIT, about 0.23%/year) force a board-level repair. For a fleet of $S$ switches, expected board-level events per year $\approx S \times \mathrm{AFR}$; for 10,000 switches at 0.23%, about 23 a year, against several pluggable swaps a day at ~300 FIT per module across the same fleet’s millions of ports. Whether CPO wins on availability depends on whether its per-port failure rate drops by more than its failure domain grows, which is what the early field data is starting to measure.⁴⁸