Learn › GPU Operations & NCCL Troubleshooting › NCCL Fundamentals
all_reduce_perf - a hands-on Linux lab on a real virtual machine.
Run your first all-reduce on two real GPUs and learn to read every column nccl-tests prints: size, count, type, redop, time, algbw, busbw, and the correctness check. Understand why bus bandwidth differs from algorithm bandwidth, why small messages are latency bound and large ones bandwidth bound, then record a baseline you can defend.
The training job used to finish overnight. This week it needs three extra hours, and nobody changed the model.
Someone on the call says the GPUs are probably talking to each other too slowly. It is a reasonable guess. Then the next question lands and the room goes quiet: how fast were they talking before?
Nobody knows. Nobody measured it when things were fine.
That is the whole problem. You cannot tell that a machine is slow if you never measured it fast. A number with nothing to compare it to is not evidence, it is a feeling.
Today you take the measurement. You will run one benchmark on a node with two GPUs, read every column it prints, and write the result to a file. In the next lab something on that node breaks on purpose, and the file you wrote is the only thing that will prove it.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every output is real, captured from a live 2x RTX 4000 Ada node. The real machine comes at the end, under Ready to practice.
Start with the word. A collective is one communication pattern that every GPU in a job takes part in at the same moment. Not one GPU asking another for a favor. All of them, together, in one operation.
All-reduce is the collective that distributed training lives on. Every GPU starts holding its own array of numbers. When the operation finishes, every GPU holds the combined result of all of them. In training, those numbers are gradients, and all-reduce is how eight or eight thousand GPUs agree on one update.
NCCL (the NVIDIA Collective Communications Library, said out loud as "nickel") is the software that makes those patterns fast. NVIDIA open-sourced it in 2015 as a small library for multi-GPU boxes. Today PyTorch, TensorFlow, and JAX all call it underneath. When a distributed job hangs or crawls, you are almost always looking at NCCL.
nccl-tests is NVIDIA's own benchmark suite for it. It ships one program per collective, each named after the collective plus _perf: all_reduce_perf, all_gather_perf, broadcast_perf, reduce_scatter_perf. Each one runs its collective at many message sizes and reports how long each took.
The node in this lesson runs NCCL 2.30.7 with CUDA 13.3, and nccl-tests 2.19.6. Versions move. The columns you are about to learn have not changed in years.
A bandwidth number means nothing until you know what it crossed. So the first command is not the benchmark. It is the map.
nvidia-smi topo -m prints the topology matrix: a grid with one row and one column per GPU, where each cell names the kind of connection between that pair. topo is short for topology, and -m asks for the matrix form.
Before you run it, commit to an answer. Two GPUs sitting in one server: do they have a private high-speed link directly between them, or does their traffic go through the CPU?
prompt: engineer@gpu-node-0:~$ answer: nvidia-smi topo -m output: GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PHB 0-7 0 N/A GPU1 PHB X 0-7 0 N/A
Legend:
X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks hint: The tool is nvidia-smi, the subcommand is topo, and the flag for matrix form is -m.
Read the grid first. Where row GPU0 meets column GPU1, the answer is PHB. The legend spells it out: a connection that traverses PCIe and a PCIe Host Bridge, which is typically the CPU. Every byte these two cards send each other goes out over PCIe, through the bridge, and back.
Now read what is missing. The legend defines NV# as a bonded set of NVLinks, NVIDIA's direct card-to-card link. No cell in this matrix says NV anything. This node has no NVLink. RTX 4000 Ada does not offer it.
That is not a defect, and it is not a simulation. It is the layout most engineers actually meet outside the biggest training clusters: a couple of GPUs sharing a bridge. Every skill here transfers to an NVLink machine. The ceiling is simply lower, which makes the ceiling easier to see.
Two more columns pay off later. CPU Affinity 0-7 names the CPU cores closest to both cards, and NUMA Affinity 0 says both sit in the same memory domain. On bigger machines those two columns are where mysterious slowdowns hide.
{ "height": 380, "caption": "Two cards, one path. Click each box, then flip the node type.", "nodes": [ { "id": "gpu0", "label": "GPU 0", "kind": "host", "x": 0, "y": 170, "detail": "NVIDIA RTX 4000 Ada Generation with 20475 MiB of memory, at PCI address 0000:00:02. Every benchmark in this lesson calls it rank 0." }, { "id": "phb", "label": "PCIe host bridge (CPU)", "kind": "nic", "x": 400, "y": 30, "detail": "The bridge the topology matrix calls PHB. Traffic leaves GPU 0 over PCIe, crosses the bridge, and arrives at GPU 1. NCCL builds a model of this machine at startup and rates this path at 24.0 GB/s." }, { "id": "nvl", "label": "NVLink bridge", "kind": "net", "x": 400, "y": 300, "detail": "NVLink is a direct card-to-card link that skips the host bridge entirely. The topology matrix writes it as NV followed by a count, such as NV4 for four bonded links. RTX 4000 Ada does not have it, so no NV code will ever appear on this node." }, { "id": "gpu1", "label": "GPU 1", "kind": "host", "x": 820, "y": 170, "detail": "The identical second card at 0000:00:03, listed as GPU1 in the matrix. Every benchmark in this lesson calls it rank 1." } ], "edges": [ { "from": "gpu0", "to": "phb", "label": "PCIe", "kind": "inband" }, { "from": "phb", "to": "gpu1", "label": "PCIe", "kind": "inband" }, { "from": "gpu0", "to": "nvl", "label": "not present here", "kind": "plain" }, { "from": "nvl", "to": "gpu1", "kind": "plain" } ], "toggle": { "label": "Node type:", "on": "This node: PHB over PCIe", "off": "A node with NVLink", "dimOn": ["nvl", "e:gpu0-nvl", "e:nvl-gpu1"], "dimOff": ["phb", "e:gpu0-phb", "e:phb-gpu1"] } }
Click both middle boxes, then flip the node type. Notice what does not change: the cards. Same GPUs, same benchmark, a different road between them. The road is what you are about to measure.
NCCL inspects the machine when a job starts and builds its own model of it. On this node that model rates the path between the two GPUs at 24.0 GB/s. You will read that number straight out of NCCL's debug log in a later lesson.
A model is a promise about the road, not about the trip.
>>> A little under. A rated speed is a ceiling, never a promise: protocol overhead, memory copies, and the collective's own bookkeeping each take a cut. The useful question is how big the cut is on this machine, and guessing does not answer it. Measuring does.
Time to run the benchmark. all_reduce_perf takes five flags that matter, and each one is a single letter.
-b is the begin size: the smallest message to test. -e is the end size: the largest. -f is the step factor, how much the size grows between tests, so -f 2 doubles it each time. -g is how many GPUs to use. -w is the number of warmup runs, throwaway passes that get everything loaded so the timed runs are not measuring startup.
Sizes take a letter suffix, so 256M means 256 megabytes.
Here is the command, read out loud: start at 8 bytes, double until 256 megabytes, use both GPUs, throw away one warmup run.
all_reduce_perf -b 8 -e 256M -f 2 -g 2 -w 1
Before you press Enter, commit to one more answer. A tiny 8 byte all-reduce and a 256 megabyte all-reduce: will they report anything like the same bandwidth?
prompt: engineer@gpu-node-0:~$ answer: all_reduce_perf -b 8 -e 256M -f 2 -g 2 -w 1 output:
#
#
8 2 float sum -1 8.33 0.00 0.00 0 8.10 0.00 0.00 0 16 4 float sum -1 8.09 0.00 0.00 0 8.02 0.00 0.00 0 32 8 float sum -1 8.46 0.00 0.00 0 8.47 0.00 0.00 0 64 16 float sum -1 8.03 0.01 0.01 0 8.05 0.01 0.01 0 128 32 float sum -1 7.98 0.02 0.02 0 8.25 0.02 0.02 0 256 64 float sum -1 7.97 0.03 0.03 0 8.14 0.03 0.03 0 512 128 float sum -1 8.16 0.06 0.06 0 8.00 0.06 0.06 0 1024 256 float sum -1 8.14 0.13 0.13 0 8.28 0.12 0.12 0 2048 512 float sum -1 8.00 0.26 0.26 0 8.46 0.24 0.24 0 4096 1024 float sum -1 8.50 0.48 0.48 0 8.16 0.50 0.50 0 8192 2048 float sum -1 8.85 0.93 0.93 0 8.84 0.93 0.93 0 16384 4096 float sum -1 10.50 1.56 1.56 0 9.24 1.77 1.77 0 32768 8192 float sum -1 10.29 3.19 3.19 0 10.21 3.21 3.21 0 65536 16384 float sum -1 13.30 4.93 4.93 0 13.38 4.90 4.90 0 131072 32768 float sum -1 19.23 6.82 6.82 0 19.20 6.83 6.83 0 262144 65536 float sum -1 30.40 8.62 8.62 0 30.25 8.67 8.67 0 524288 131072 float sum -1 49.61 10.57 10.57 0 49.52 10.59 10.59 0 1048576 262144 float sum -1 71.16 14.74 14.74 0 71.15 14.74 14.74 0 2097152 524288 float sum -1 115.03 18.23 18.23 0 113.07 18.55 18.55 0 4194304 1048576 float sum -1 207.11 20.25 20.25 0 207.48 20.22 20.22 0 8388608 2097152 float sum -1 397.03 21.13 21.13 0 397.40 21.11 21.11 0 16777216 4194304 float sum -1 785.14 21.37 21.37 0 782.02 21.45 21.45 0 33554432 8388608 float sum -1 1551.75 21.62 21.62 0 1552.97 21.61 21.61 0 67108864 16777216 float sum -1 3085.89 21.75 21.75 0 3083.85 21.76 21.76 0 134217728 33554432 float sum -1 6139.29 21.86 21.86 0 6135.19 21.88 21.88 0 268435456 67108864 float sum -1 12157.1 22.08 22.08 0 12137.1 22.12 22.12 0
#
# hint: The five flags in order: -b 8 for the begin size, -e 256M for the end size, -f 2 to double, -g 2 for both GPUs, -w 1 for one warmup run.
There it is. Twenty-six sizes, one row each, and a machine that a moment ago was an assumption is now a set of numbers.
Work across one row and every column earns its keep.
size is how many bytes each GPU holds for this test. count is the same amount expressed in elements: 8 bytes divided by 4 bytes per float is 2 elements. type is float, the default 4-byte floating point number. redop is the reduction operation, the arithmetic used to combine values, here sum. root is -1, meaning not applicable, because all-reduce has no owner: every rank contributes and every rank receives.
time (us) is microseconds for one collective, averaged over the timed runs. A microsecond is one millionth of a second. algbw and busbw are two different bandwidth numbers in gigabytes per second, and the next step is entirely about why there are two.
#wrong is the column to check first and quote never. The benchmark verifies the answer every rank ended up with. 0 means every rank got the correct result. Anything else means the collective produced wrong numbers, and a speed measurement of a wrong answer is worth nothing.
The table then repeats itself. The left half is out-of-place, where input and output live in separate buffers. The right half is in-place, where the result overwrites the input, which is what training frameworks normally do. On this node the two halves land within a percent of each other, so either half tells the same story.
Now put the first row beside the last row:
# size count type redop root time algbw busbw #wrong
8 2 float sum -1 8.33 0.00 0.00 0
268435456 67108864 float sum -1 12157.1 22.08 22.08 0The message grew by a factor of 33,554,432. The time grew by a factor of about 1,460.
That gap is the entire mental model of collective performance, and you just generated the evidence for it yourself.
Stop for a second, because you have just crossed a line that most people never cross.
"The cluster feels slow" is a complaint. "This node all-reduces 256 megabytes at 22.08 GB/s of bus bandwidth, and last month it did the same job at the same rate" is an argument. One of those gets an engineer paged. The other gets a fix.
You now know how to produce the second kind of sentence. Everything after this is about reading it more precisely.
algbw is algorithm bandwidth, and it is plain division: the message size divided by the time it took. It answers "how fast did my buffer get handled", from the caller's point of view.
busbw is bus bandwidth, and it estimates how much traffic actually crossed the wire. It is algbw multiplied by a correction factor, and the factor depends on which collective ran and how many ranks took part.
For all-reduce that factor is 2(n-1)/n, where n is the number of ranks. The shape of it comes from how the work is really done. The data goes around the group twice: once to combine the values, once to hand the finished result back out. And each rank only sends the (n-1)/n of the buffer that is not already its own.
Put 2 into that formula. 2(2-1)/2 is exactly 1. On a two-GPU node the correction does nothing at all, which is why every row of your sweep shows algbw and busbw as the same number.
Put 8 into it and the two columns separate: 2(8-1)/8 is 1.75, so bus bandwidth reads 1.75 times higher than algorithm bandwidth on an eight-GPU node.
That is why busbw is the number engineers compare against hardware. Algorithm bandwidth drifts as you add ranks even when the wire never changed. Bus bandwidth stays comparable, so it is the one you put next to a rated link speed.
>>> The factor is 1 here and only here. Two ranks put 2(2-1)/2 at exactly 1, so the correction multiplies by one and the columns match. Add ranks and they pull apart. The next challenge proves the factor is real by running a collective whose factor is not 1.
Different collective, different factor. All-gather does no arithmetic: every rank simply ends up holding a copy of everyone's data. Its correction factor is (n-1)/n, which at two ranks is 0.5.
So the theory makes a hard prediction: in an all-gather run on this node, busbw should be exactly half of algbw in every row. Test it.
Run the all-gather benchmark from the same suite over a shorter range: begin at 8 megabytes, end at 128 megabytes, step by a factor of 4, both GPUs, one warmup run. No command is shown for this one.
prompt: engineer@gpu-node-0:~$ answer: all_gather_perf -b 8M -e 128M -f 4 -g 2 -w 1 output:
#
#
8388608 1048576 float none -1 218.87 38.33 19.16 0 218.37 38.41 19.21 0 33554432 4194304 float none -1 832.08 40.33 20.16 0 832.17 40.32 20.16 0 134217728 16777216 float none -1 3272.37 41.02 20.51 0 3252.65 41.26 20.63 0
#
# hint: The binary is named after its collective: all_gather_perf. Same five flags, with 8M as the begin size, 128M as the end size, and 4 as the factor.
Check the arithmetic against the prediction. Bottom row: 41.02 and 20.51, and half of 41.02 is 20.51. Top row: 38.33 and 19.16. Every row, exactly half, exactly as (n-1)/n said it would be.
The redop column tells you why the factor differs at all. It reads none here, not sum: all-gather concatenates data instead of combining it, so i
Practice Your First All-Reduce in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.