Trust the FEC, not the thermometer

The most precise number in the module

Every modern transceiver holds an estimate of the link’s signal-to-noise ratio (SNR) in its memory map, the register file that the host reads. The module’s own digital signal processor (DSP) computes it from the amplitude histogram of the electrical eye. The register reports it in units of one two-hundred-and-fifty-sixth of a decibel. That is about four thousandths of a decibel per step. The number is satisfying to read, and it looks like instrumentation.

Now read the tolerance that the specification attaches to it. The Common Management Interface Specification (CMIS) applies to every QSFP-DD and OSFP module in production. Its requirement applies at the vendor-specified wavelength and over the specified temperature and voltage. There, the accuracy of the reported signal-to-noise and level-transition observables shall be better than plus or minus 3 decibels.

The register resolves four thousandths of a decibel, but the accuracy is only plus or minus three decibels. That is 768 steps for each three decibels of permitted error. The same specification sets these other tolerances:

• Transmit and receive optical power: plus or minus 3 decibels.
• Bias current: plus or minus 10 percent.
• Temperature: plus or minus 3 degrees, which CMIS inherits from the older SFF documents.

This is not a scandal, and it is not sloppiness. A monitor costs this much when it must operate across every vendor, every wavelength, the full commercial temperature range and a supply rail that moves. And the device has only a fraction of a watt to spend on measuring itself. But this tolerance answers a question that people ask often, and usually hopefully. Can the network act on this telemetry? In absolute terms, no.

A receive power reading can be wrong by 3 decibels in either direction. That range is a factor of four in optical power. You cannot set a fleet-wide threshold on that, and you cannot compare the modules of two vendors with it. You can watch one lane drift against its own history. That is real and useful, and it is the only use that any of these numbers support.

What the module is actually watching

The control loops are more interesting than the registers, so it is useful to be concrete about what the module measures.

For a given bias current, the output of a laser is not constant. It decreases with temperature and with age. The module compensates continuously with a monitor photodiode, which catches light that leaks from the back facet. That photocurrent is a proxy for output power. An automatic power control loop adjusts the bias to hold the front-facet power at the value in the datasheet.

Over the life of the part, the bias current that the loop asks for slowly increases. That increase is the module’s own record of its laser aging. It is the single most useful thing in the memory map, because it is a trend and not an absolute value.

A Mach-Zehnder modulator needs a second loop. Its transfer function is a raised cosine, and the operating point sits on a slope of that curve. Thermal drift and charge migration move the point. If nothing holds the point, the extinction ratio collapses.

The classical solution adds a small low-frequency dither to the bias. The module then watches the harmonics of that tone in the monitor photodiode. At quadrature (the midpoint of the slope), the second harmonic of the dither disappears. When it appears again, it shows that the point drifted. Its sign shows the direction. So the module runs a lock-in amplifier on itself, continuously, and reports the resulting bias voltage as one more monitor.

Then add the other loops and sensors:

• A thermoelectric cooler loop, where the wavelength must stay constant.
• One temperature sensor for each thing that matters.
• Bias and power for each of eight lanes.
• The supply rails and the module’s own conversions.

The analog inventory is therefore substantial. A few digital-to-analog converters drive bias, modulation and bias points. A multiplexed analog-to-digital converter steps through a dozen or more monitor channels.

The chip that stopped existing

Which part does all this work? The honest answer is that it was a dedicated part before, but it is not any more.

There were two families of purpose-built module management ASSPs (application-specific standard products). Both put the control loops in hardware:

• An on-chip amplifier that closed the power loop.
• Temperature-compensation lookup tables.
• Calibrated converters.
• A diagnostics map that the silicon served directly.

One of them measured temperature to about a degree, typical. That is three times better than what CMIS asks of a module today. It drew under ten milliwatts to do it.

Both families stop at SFP+ and the SFF-8472 diagnostics map. Neither reached CMIS. No replacement shipped.

The reason is in the specification, not in the analog circuits. CMIS asks a module to do these tasks:

• Serve a banked, paged register file.
• Run a module state machine and a data path state machine.
• Accept a firmware image from the host over a command-block messaging layer, with commands to start, abort and complete a download.

A part that must accept new firmware in the field, and survive the update, is not running control loops. It is running an embedded application. Embedded applications use microcontrollers.

So the socket moved. Examine what general-purpose microcontrollers now advertise. None of them says “optical” anywhere in the silicon, but the shape of this application is clear:

• A dual-address two-wire slave, which is the old A0h and A2h pair in hardware.
• Dual-bank flash, so the module can replace its firmware while it continues to run.
• An unusual number of digital-to-analog converters.
• A management data interface next to the two-wire bus, because the host can use either.
• A package four millimeters square.
• Pins that tolerate 1.8 volts, so the microcontroller can communicate with the signal-processing chip without level shifters.

This gives the best single line in this whole story. One 800G PAM4 DSP lists an embedded CPU for real-time control among its own features. Immediately after that, it lists a serial and two-wire slave interface to communicate to the module microcontroller. Many people think of the DSP as the brain of a module. In this design it is a peripheral. The brain is a commodity part that runs firmware.

Five decades of cadence

Now trace the numbers from the module to the network. At each step, the timescale becomes longer.

The module samples its own observables at an interval that it does not publish. CMIS says only that the interval is vendor-defined and typically sub-second. The host reads the observables over a two-wire bus at 400 kilohertz by default. On that bus, a 128-byte page costs about three milliseconds of pure bus time. A change of the mapped page can block the bus for up to ten more.

Multiply by the ports on a switch, and the polling cycle becomes visible. A 64-port box that reads five pages per port uses about a second of bus time. If every page change takes its worst case, the box uses about four seconds.

That explains the default. The most widely deployed open network operating system polls diagnostics every 60 seconds. It records the interval that it actually achieved, because the target and the result differ. Its design states that it does not try to handle rapid link flaps. Its streaming telemetry refuses sample intervals below one second. The vendor conformance suite samples optics at ten.

Why a minute

A monitor read is a page read on a slow bus, and a switch has to walk every port. Push the port count up or the bus speed down and the polling cycle collides with the interval it is supposed to fit inside.

Ports on the switch

64

Pages read per port

5

Bus clock

400 kHz default1 MHz option

Page change cost

No hold-offWorst case

4.3 s

to walk every port once

7%

of a sixty-second budget

Page reads per cycle

320

64 ports x 5 pages

One page read

2.95 ms

at 400 kHz

Pure bus time

0.94 s

before any hold-off

Worst-case hold-off

10 ms

per page-mapping change

There is headroom here, but not much of it, and every new diagnostic page a module advertises spends some.

Bus speeds and the ten-millisecond page-change hold-off are from the CMIS management interface timing tables. The per-page arithmetic is ours: a 128-byte page plus three bytes of addressing, nine bits per byte on a two-wire bus. The sixty-second budget is the default diagnostic poll interval in the most widely deployed open network operating system.

That cadence has a cost beyond latency. CMIS flags latch and clear on read. So a poll tells you that something happened, but not how many times. The design document of the operating system gives an example: a flag that toggles six times between two polls shows a change count of two. The module knows in milliseconds. The network learns in minutes, and it learns less than what happened.

None of this is new, and that is oddly comforting. The entity sensor MIB of 2002 already had an object with one purpose: to tell you how often a sensor value refreshes. One of its enumerated values means that the agent does not know. Twenty years later, the honest answer is the same, but the mechanism is better. gNMI, the streaming management interface, puts a timestamp on every notification and lets the client choose its own interval.

What operators actually act on

If the analog monitors are trending instruments and the cadence is a minute, what closes a loop?

The answer is forward error correction (FEC). Every high-speed link runs FEC, and the decoder keeps these counts:

• Corrected codewords.
• Symbol error distributions.
• Uncorrectable codewords, which are real loss.

That counter is fast, and it is exact, not approximate. It sits at the end of the full chain, so it integrates every impairment that the analog monitors describe individually. The modern memory map does carry pre-FEC bit error rate and symbol error weight as observables, so the module can report its own view. But the host has the same counters, at ASIC speed, with no two-wire bus in the path.

So the deployed control loop is short and plain:

• Error counters cross a threshold.
• The system disables or drains the link.
• The routing protocol redistributes traffic across the remaining members.

The optical telemetry is not in that loop. It is in the maintenance loop. A technician gets a ticket. The bias trend and the power history then tell the technician which action to take: replace a module, clean a connector, or inspect the fiber.

The interesting failures occur in the gap between those two loops. The consortium that promotes linear drive (modules with no DSP) wrote the clearest description of it. In a link with multi-path interference, both modules report that their signal levels are fine. But the host reports an SNR lower than it should be. Nothing in the module is wrong.

To find the cause, a technician must take an optical time-domain reflectometer to the fiber. With two healthy modules on one faulty link, the module data cannot locate the fault. This failure mode increases as the module loses its own signal processing.

Where the boundary goes when the module dissolves

Two architectures now in production remove parts of this picture. They work in opposite directions, but they have the same consequence.

Linear drive removes the module’s DSP. The power saving is real, and companies state it in filings. One switch vendor puts the interconnect saving at roughly 60 percent at 1.6T. But the module’s own view of the signal leaves with the DSP. Error counting moves fully to the host. The media-side and host-side split used to tell you which end owned a problem, and that split becomes thinner.

A hyperscaler published its position on linear drive at 200G per lane. It prefers the half-retimed variants. The reason it names is qualification and interoperability burden, not link budget. Its justification shows the real point: a specification can define a transmit-retimed interface well enough to guarantee interoperability. What the half-retimed design restores is a testable boundary.

Co-packaging removes the module completely, and the field data is now public. A hyperscaler evaluated a 51.2T co-packaged switch. It monitored four observables per channel at a five-minute cadence:

• Laser bias.
• Transmit power.
• Receive power.
• SNR.

That cadence is five times slower than the sixty-second poll of the pluggables that it replaces. The evaluation judges link health from the distribution of the highest non-zero FEC bin across 128 ports, with data from the switch ASIC.

The serviceability sentence is the one to keep. With co-packaged optics, the optics is no longer a field-replaceable component. A failure in an optical engine means that the operator must replace the full chassis. The mitigation is that the lasers stay pluggable on the front panel. The managed memory map also survives there, because the external laser has its own management agreement.

So the most integrated optics deployed at scale gets slower monitoring than the pluggable it replaced. The monitoring goes through a vendor-defined internal path, and the actionable signal comes from the host. Eighteen months into the co-packaging transition, the socket agreements list a management interface in scope. But no published document defines it.

What to take away

1. Nothing in the memory map of the module is decision-grade in absolute terms. Power and SNR have plus or minus 3 decibels, and bias has plus or minus 10 percent. Trend them against their own history, never against a fleet threshold.
2. The precision of the register is not the accuracy of the measurement. An observable with a resolution of four thousandths of a decibel and a guarantee of plus or minus three gives 768 steps for each three decibels of permitted error. The accuracy does not support those steps.
3. Field firmware update, not analog requirements, removed the dedicated management chip. The module’s DSP is now a slave to the microcontroller that replaced that chip.
4. Cadence spans five decades, from a three-millisecond page read to the five-minute poll of a co-packaged switch. Latched flags mean that a poll counts fewer events than happened.
5. The loop that acts goes from FEC counters to link state to routing. Optical telemetry feeds maintenance, not traffic. Each architecture that moves signal processing out of the module makes that division sharper and attribution harder.

The general lesson is about where a measurement lives. Telemetry is most useful at the boundary where you can act. The industry continues to move the measurement toward the host, because there it is cheap, fast and exact. Each move is correct locally. Together, the moves remove the boundary between the component that failed and the system that noticed.

The counter that tells you something is wrong continues to improve. The industry is quietly abandoning the instrument that tells you what to replace. No one proposed a replacement for it yet, for the time when the module is no longer a part that you can pull.