I recently started getting hands-on with GPU infrastructure. Before running any workloads on the cluster, I wanted to benchmark the network and see what the hardware was actually capable of.
The setup was two servers with eight NVIDIA H200s in each, so sixteen GPUs. I’ve spent about twenty years in ordinary IT infrastructure, but I’m relatively new to the GPU world, so a decent chunk of what follows is me getting things wrong and working out why. I’ve left those bits in. They were the useful part.
Why the network matters at all
Training a model across multiple GPUs only works if the GPUs stay in sync.
Each one takes a different batch of data and calculates how the model’s weights should change. Then all of them have to combine those calculations, so everyone ends up holding an identical model. If they didn’t, sixteen GPUs would slowly drift into training sixteen slightly different models.
That combining step is called an all-reduce. Sixteen people each work a different section of a maths problem, and at the end of every page everyone adds up their totals and shares the result so all sixteen start the next page from the same number.
It happens on every training step. Thousands of times per run. So if it’s slow, your very expensive GPUs finish their arithmetic in a few milliseconds and then sit there doing nothing while they wait for each other.
Two separate networks are involved. NVLink connects the GPUs inside a single server , dedicated wiring on the motherboard, very short distances. InfiniBand connects the servers to each other. It looks like Ethernet cabling but it’s a different technology built for low delay.
I wanted to know what each one is actually worth, in seconds, rather than what the spec sheet says.
The benchmark
nccl-tests is NVIDIA’s tool for this. NCCL is the library that moves data between GPUs, PyTorch and everything else use it underneath, so when your training code syncs gradients, NCCL is what’s doing the work.
The benchmark doesn’t train anything. There’s no model and no dataset. It puts a block of data on each GPU, tells them all to combine it, times how long that took, then makes the block bigger and does it again. It works up from 8 bytes to 8 gigabytes.
The reason for testing so many sizes is that small and large transfers fail differently. A small transfer is mostly fixed overhead: the cost of starting a transfer at all, which you pay regardless of size. A large transfer is mostly about capacity. It’s the difference between posting a postcard and shipping a tonne of bricks, both take a day, but for completely different reasons. A network can be good at one and bad at the other, so a single headline number hides half of what’s going on.
I ran four configurations, all with one process per GPU so they’d be comparable:
| Servers | GPUs | Disabled | |
|---|---|---|---|
| A | 1 | 8 | nothing |
| B | 1 | 8 | NVLink |
| C | 2 | 16 | nothing |
| D | 2 | 16 | InfiniBand |
B and D are the controls. Turning something off and measuring again is the only way to prove what it was contributing.
One thing to be clear about with D: disabling InfiniBand only affects traffic between the two servers. NVLink inside each server carried on working normally. So it isn’t a test of “no fast networking at all,” it’s a test of “fast inside each box, slow between boxes.”
The results
Time to complete a single 8 GB all-reduce:
| Configuration | Time | Slowdown |
|---|---|---|
| 8 GPUs, NVLink | 31.31 ms | – |
| 16 GPUs, NVLink + InfiniBand | 33.47 ms | 1.07x |
| 8 GPUs, NVLink off | 142.51 ms | 4.6x |
| 16 GPUs, InfiniBand off | 6,412 ms | 192x |
That last row is the one that surprised me. The same operation that takes 33 milliseconds takes 6.4 seconds when the link between servers drops to ordinary TCP.
Losing InfiniBand costs 192x. What makes it that severe is that NVLink was still running at full speed inside both servers. Only the connection between them degraded. But all sixteen GPUs have to finish together, so however fast the other links are, everybody waits for the slow one.
Losing NVLink costs 4.6x. Turning it off pushes data through the server’s main memory and out over PCIe instead of GPU to GPU directly. 142 ms against 31 ms. Slower, clearly, but PCIe is still a real path. It’s a loss rather than a collapse.
A quick warning about reading those two numbers side by side, because I nearly drew the wrong conclusion from them myself. They are not saying InfiniBand is more important than NVLink, and they’re certainly not saying it’s faster. NVLink on this hardware moves 900 GB/s per GPU (bidirectional); InfiniBand, with a dedicated 400 Gbps card per GPU, gives closer to 50 GB/s per GPU (unidirectional). NVLink is far and away the faster of the two.
What those relative slowdowns actually measure is how bad each fallback is. NVLink falls back to PCIe, which is a genuine high-speed bus and still manages 105 GB/s. InfiniBand falls back to TCP over Ethernet, which manages 2.5 GB/s. The 192x figure is mostly a statement about how poorly standard TCP handles collective GPU operations, rather than a measure of InfiniBand’s absolute speed.
Scaling to a second server cost about 7%. Going from 8 GPUs on one machine to 16 across two added roughly two milliseconds to the sync. Two things changed at once here, a network hop got added and the GPU count doubled, so this is the cost of scaling out generally, not of the network on its own. Either way it’s a good trade, since you’re paying 7% more on the sync while getting through twice as much data per step.
Small transfers are a different story
The 8 GB number flatters things. Here’s the same comparison at 1 MB:
| Time | |
|---|---|
| 8 GPUs, one server | 26.5 µs |
| 16 GPUs, two servers | 88.0 µs |
3.4x slower, against the 7% we saw at 8 GB.
Below a few megabytes the time is mostly latency, and bandwidth can’t do anything about latency. Crossing between servers adds a fixed delay that you pay on every single transfer. If your training syncs frequently in small chunks, that’s the number that governs you rather than the headline one.
There’s a related pattern in the TCP numbers. At 1 MB, TCP was 17x slower than InfiniBand. At 8 GB it was 192x. TCP is relatively less terrible at small sizes because everything is latency-bound down there. The gap opens up as messages grow, because TCP hits a wall at around 2.5 GB/s and stays there while InfiniBand keeps climbing to 480.
Three things I got wrong
I compared two runs that weren’t comparable
My first attempt ran the single-server test as one process driving all eight GPUs, and the two-server test as sixteen processes with one GPU each.
The total process count differing is fine, that’s just a consequence of having different numbers of GPUs. The problem was GPUs per process: eight in one case, one in the other. A single process driving eight GPUs has to issue commands to them one after another, and that costs time on every operation.
You could see it in the output. Every message size below about a megabyte took roughly the same 45 microseconds. 8 bytes took 47 µs. 4 KB took 47 µs. A network doesn’t take the same time for 8 bytes as for 4 KB, so that flat floor was clearly the process rather than the wire.
When I re-ran the single-server test with one process per GPU, the 1 MB bandwidth went from 36.79 GB/s to 69.20. Nearly double.
My first draft of this post said small messages cost 33% going multi-server. The real figure is 3.4x. I’d been wrong in the direction of making the network look better than it is, which is the worse direction to be wrong in.
The large-message numbers weren’t affected. 45 microseconds against a 31 millisecond operation is nothing.
I compared two bandwidth figures that had been calculated differently
The benchmark prints two bandwidth columns for every measurement and they don’t agree with each other. Working out why took me a while, and it’s easier to follow if you first know what’s actually crossing the wires in a real training job.
What gets sent. Say your model has 7 billion parameters. Each GPU is working on a different batch of data, and produces one gradient per parameter, a number saying which way that parameter should move. So each GPU ends up holding 7 billion numbers, which in the usual 16-bit format is about 14 GB.
Those 14 GB are what gets combined. Worth noticing: that’s the size of the model, not of your data. Feeding in a bigger batch doesn’t make the sync bigger. A bigger model does.
So every training step, all sixteen GPUs are sitting there with 14 GB of gradients each, and they need to be added together with everybody receiving the total.
The first number: how fast did that job finish?
14 GB divided by however long it took. That’s algorithm bandwidth, and it’s the number that tells you how much of your step time went on waiting for the network. In the benchmark, 8 GB in 31.31 ms works out to 274 GB/s.
The second number: how much traffic did that actually generate?
Here’s the part I hadn’t appreciated. Combining 14 GB across eight GPUs doesn’t mean each GPU sends 14 GB. It sends considerably more.
Picture the eight GPUs in a circle. GPU 1 hands its numbers to GPU 2, which adds its own and hands the running total to GPU 3, and so on all the way round. When it arrives back at the start you have the answer, but only one GPU is holding it. So it goes round the circle a second time to tell everybody else.
Two laps. So each GPU’s link carried roughly twice the payload: about 24.5 GB, not 14.
Bus bandwidth is the figure that accounts for that. Same job, same elapsed time, but measuring what went over the wire rather than what the application asked for:
274 GB/s × 1.75 = 480 GB/s
Why have both? Because they answer different questions.
- Will my training be slow? -> algorithm bandwidth. It’s your actual sync time.
- Am I getting what this hardware is capable of? -> bus bandwidth. NVLink’s spec is 900 GB/s per GPU, and the only way to know how close you are is to measure what’s really on the wire.
Where 1.75 comes from. It’s slightly under 2 because on each lap, one GPU already holds the data and doesn’t need it sent. With 8 GPUs, 7 out of 8 do:
scaling factor = 2 × (N − 1) / N where N is the number of GPUs
| GPUs | Working | Scaling factor |
|---|---|---|
| 8 | 2 × 7/8 | 1.75 |
| 16 | 2 × 15/16 | 1.875 |
And there’s the problem: that factor is different for different GPU counts. A bigger circle means more passing, which means a bigger factor.
My single-server run reported 480.15 GB/s bus bandwidth. My two-server run reported 481.37. I looked at that and wrote a paragraph about the network scaling perfectly.
But those two numbers had been scaled by different amounts before I ever saw them. The actual times were 31.31 ms and 33.46 ms, so the two-server run was 7% slower. The larger scaling factor on the 16-GPU calculation had quietly swallowed the difference.
Bus bandwidth is fine for comparing against a hardware spec sheet, or for comparing two runs that have the same number of GPUs. Across different GPU counts you need to use wall-clock time. That’s why the InfiniBand-versus-TCP comparison earlier holds up, incidentally, both of those runs had 16 GPUs, so both were scaled by the same amount.
I measured nothing at all and didn’t notice
One early run reported a bus bandwidth of exactly 0.00. My first thought was that the hardware was broken.
What had actually happened was that I’d started eight separate jobs, each using one GPU, none of them aware the others existed. I thought I was starting one job that used eight GPUs together. Since no GPU ever sent anything to another GPU, there was nothing to measure.
The reason it showed up as zero rather than as an error goes back to the scaling factor from the previous section. Bus bandwidth is the raw speed scaled up by how many GPUs are passing data round the circle. With only one GPU, there is no circle. Nothing gets passed to anybody. Run the numbers and the scaling factor comes out as 2 × (1−1) / 1, which is zero, and zero times anything is zero.
The benchmark was being completely honest. I just didn’t understand what it was telling me.
What makes this worth writing about is that the test finished successfully. It printed a full page of results: message sizes, timings, bandwidths, all present, all plausible-looking. Nothing said error, nothing warned me. If I hadn’t happened to notice that one column reading 0.00 I would have written the numbers down and believed them.
The check that would have caught it takes about two seconds. Before its results table, the benchmark prints one line per GPU taking part. Count them. If you asked for eight and see one, or see eight lines that all claim to be GPU number zero, the test didn’t do what you thought.
The underlying lesson is that starting eight programs doesn’t make them a team. They need something to introduce them to each other, usually MPI or torchrun. Without it, each one quietly assumes it’s the only one there and gets on with doing nothing useful.
The thing worth checking on your own cluster
A server usually has several network connections. The fast InfiniBand ones, and an ordinary Ethernet one for logging in and general traffic. When a job starts, NCCL decides which to use.
If it picks the ordinary one, nothing goes wrong. No error, no warning. The job starts, trains, finishes. It’s just 200 times slower on every sync, and the only symptom is that training feels slow for no obvious reason. Usually it’s a configuration setting pointing at the wrong network card.
NCCL will tell you what it picked, but only if you ask it to be verbose:
NCCL_DEBUG=INFO your-training-command 2>&1 | grep -o "NET/[A-Za-z0-9/]*" | sort -uNET/IB means InfiniBand. NET/Socket means ordinary networking, and you should go and find out why.
Mine showed NET/IB/0/GDRDMA through NET/IB/7/GDRDMA, one line per InfiniBand card, so all eight were in use. GDRDMA is worth knowing about separately. Normally, getting data out of a GPU means copying it into the server’s main memory first and then out to the network card. GPUDirect RDMA lets the network card read GPU memory directly and skip that step. It’s a separate thing from InfiniBand and has to be configured, so it can be missing on an otherwise working setup. Seeing it in the log is a confirmation rather than something to take for granted.
One number in the output that lies
At the bottom of each run, the benchmark prints a summary:
# Avg bus bandwidth : 146.751
It looks like the answer. It’s an unweighted average across all 31 message sizes, including the 8-byte ones, and those score almost zero because their time is entirely startup overhead. That drags the average right down. My real bus bandwidth at a useful message size was 480 GB/s. The summary line said 147.
Quote specific message sizes instead, and show how the number changes across them.
What I didn’t test
Worth being explicit, because I nearly published a few claims I hadn’t earned.
I only used two servers. So when I say scaling out cost 7%, that was going from one server to two. With sixteen or sixty-four servers a proportionally larger share of the traffic has to cross the network between them, and nothing I measured predicts what happens there. It would be worse. How much worse, I don’t know.
I measured the network on its own, with no model and no data pipeline. A real training job with a degraded network would show something related to these numbers but not the same as them.
I only tested all-reduce. There are other GPU-to-GPU operations, gathering data without combining it, scattering it out, broadcasting from one GPU to all, and they load the network differently.
And I didn’t test anything about how you should split a model across GPUs. There are several strategies and they sync at very different rates; standard practice is to keep the chatty ones inside a single server. These numbers are consistent with why that’s the advice, but a network benchmark isn’t a training job, so treat that as reasoning rather than something I demonstrated.
If there’s one thing to take away: check which network NCCL chose before you start a long job. It fails silently, and two orders of magnitude is a lot to lose to a configuration setting.
All four benchmark runs, the commands that produced them, can be found here.
