Google got about 192 Gb/s out of a single 200 Gb/s network interface while streaming data between TPU hosts. With two interfaces and transfers split across them, it measured about 350 Gb/s. Those are peak figures from Google’s July lab tests on Ironwood, published November 9.
What needs that much bandwidth before a model answers a question? In this setup, one group of chips reads the prompt and builds a key-value cache, the intermediate state the model needs to generate a response. A different group generates the tokens. It needs that cache before it can start, so the handoff between hosts shows up in the time you wait for the first token.
The detail I’d take into a capacity review is how Google used the second interface. TPU Sync stripes the transfer across both NICs and accounts for which memory is local to each one. If I were sizing that path, I’d ask for the single-NIC and dual-NIC results under our expected concurrency. Adding the port speeds on a diagram gives me 400 Gb/s. It doesn’t tell me how long a request will wait for its cache.
Google’s TPU Sync engineering writeup and test configuration