Nokia ran a simulation of 2.048 million atoms on 16 NVIDIA H200 GPUs. It managed 821 timesteps per second. After the engineers changed how the messaging software used the network interfaces, the same GPUs ran 1,105 timesteps per second. That’s about 35% more simulation work every second.
The test split the atoms across two servers connected by Ethernet. Each GPU calculated what happened in its part of the simulation, then exchanged data about atoms near the boundary with its neighbors. A GPU that finishes its own calculations can still spend time waiting for another GPU’s boundary data.
Nokia attributes the improvement to the way MPI, the message-passing interface, and its UCX communication layer accessed the network adapters. The write-up leaves out the before-and-after settings, so there’s no specific tuning knob to copy.
For an AI training cluster, I’d borrow the test method. Keep the model, batch size and GPU count fixed. Record which network adapters carry the traffic, change the interface selection, then compare training step times. That gives you a way to tell whether a busier link actually got the job done sooner.