When a switch tells its neighbor to pause, packets are still coming. The pause has to travel upstream, and the packets already on the fiber have to arrive somewhere. On a long link, a switch can run out of room even when the neighbor does exactly what it was asked to do.
That’s a practical problem for Meta’s AI training network. Its backend aggregation layer connects GPU fabrics across buildings and campuses, with some links stretching tens of kilometers. The switch needs to reserve enough buffer space, called headroom, to absorb traffic while the pause takes effect. At the same line rate, a longer fiber means more bytes in flight.
In its October 12 engineering writeup, Meta describes how its switch software, FBOSS, sizes that reserve. It times probe traffic on an uncongested queue, estimates the fiber length from the round trip, and allocates headroom for the measured distance. The queue choice matters. Time spent waiting behind training traffic would make the fiber look longer than it is.
That leaves me with a question I’d ask after an optical circuit gets rerouted: when does the switch measure again? The ports can stay the same while the fiber between them gets longer.
Source: Meta Engineering’s OCP networking writeup, “BAG: Scale-Across Aggregation”.