A smaller average traffic load can still hide a burst that stops training.
Google’s November 19 MaxText report tested Qwen3-32B across two TPU v5p-32 slices. With inter-slice bandwidth limited to 10 Gb/s using Linux traffic control, ordinary data-parallel training took 55.11 seconds per step. Streaming DiLoCo took 5.76 seconds. This was a bandwidth-shaped test, not a reported intercity deployment.
DiLoCo lets compute groups train locally before exchanging model updates. The vanilla version sends the full update every 65 steps in this test. It reduced communication but still paused for that periodic transfer, averaging 7.55 seconds per step.
Streaming divides the model into 65 fragments and synchronizes one per step. Host CPU workers exchange the fragment updates while TPUs keep computing. Google’s measured traffic fell from 4,258.8 GB to 67.0 GB over 65 steps versus data parallelism.
The MaxText engineering guide makes the distinction explicit: vanilla and streaming DiLoCo move the same total bytes. Streaming changes when they move.
For network sizing, measure the synchronization burst alongside average utilization. Then validate training convergence on your workload before trading synchronization frequency for a thinner link.


