Learn › GPU Operations & NCCL Troubleshooting › NCCL Troubleshooting
A training job silently lost a fifth of its throughput overnight with no error and no alert. Measure the all-reduce on two real GPUs, read the NCCL transport line, trace the downgrade to the variable that caused it, and prove the fix in GB/s.
It is 08:05 on a Tuesday. The nightly training job on gpu-node-07 finished overnight, the way it always does. Exit code 0. Nothing in the error log. No alert fired.
It also finished late.
The engineer on call already checked the obvious things. Both GPUs are present, cool, and idle. No stuck process. Nothing in dmesg, the kernel message buffer where hardware faults usually shout. Every dashboard is green.
Here is what makes this kind of incident nasty. Nothing is broken. The job ran. The math is correct. The hardware is healthy. And it is moving data between the two GPUs at about 19 percent less bandwidth than it did last week.
By the end of this lesson you will find the cause in one line of log output. That line is about forty characters long. It never appears unless you ask for it. It is the single most useful string in GPU cluster troubleshooting.
This is a practice zone. The black boxes below are a safe sandbox that checks one command at a time, and the diagrams react to you: click a box, flip the switch. The real two-GPU machine comes at the end, under Ready to practice.
Training a model on two GPUs is not two jobs. It is one job, run twice, in lockstep.
Each GPU works through its own slice of the data and produces its own gradients: the numbers that say how much every weight in the model should change. Before anyone can take a step, those two sets of numbers must be added together and handed back to both cards. Both GPUs have to end up holding the identical result.
That operation has a name. It is an all-reduce. Reduce, because many values collapse into one. All, because every participant gets the answer back, not just one of them.
The library that performs it is NCCL, the NVIDIA Collective Communications Library. Say it "nickel". Operations that involve every participant are called collectives, which is where the name comes from. Each participating process is a rank. Rank 0 drives the first GPU, rank 1 drives the second.
This happens after every training step. Thousands of times an hour. If the all-reduce gets slower, the whole job gets slower, and nothing reports an error anywhere.
NCCL came out of NVIDIA in 2015 as a single-node library for the GPUs inside one machine. Version 2, in 2017, taught it to cross machines. The library on the node in this lesson identifies itself as NCCL version 2.30.7+cuda13.3.
>>> Nothing failed. NCCL promises that the collective completes and that the answer is correct. It never promises which route it took. When the quickest route is unavailable it quietly picks the next one down and returns correct results. Alerts fire on errors, and there is no error here. That is exactly why this bug survives for weeks.
Before you benchmark anything, find out what the two GPUs are physically allowed to do. nvidia-smi is the NVIDIA management tool that ships with the driver, and its topo subcommand prints the topology: how each device connects to every other device.
Run the matrix form with nvidia-smi topo -m and read what comes back.
prompt: ops@gpu-node-07:~$ answer: nvidia-smi topo -m output: GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PHB 0-7 0 N/A GPU1 PHB X 0-7 0 N/A
Legend:
X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks hint: Type it exactly: nvidia-smi topo -m
Read the grid like a distance chart in a road atlas. Row GPU0, column GPU1 holds one word, and that word is PHB. The legend spells it out: the connection crosses PCIe, the general purpose expansion bus every card in a server plugs into, plus a PCIe Host Bridge, which is a part of the CPU itself.
So the two cards talk through the CPU. Notice what is not in that grid: NV#, the legend entry for NVLink, the dedicated high speed GPU to GPU cable NVIDIA puts on datacenter boards. It is defined in the legend and it appears nowhere in the matrix. This node has no NVLink. Most budget and mid-range multi-GPU nodes do not.
{ "height": 380, "caption": "One pair of GPUs, two possible routes. Click any box, then flip the transport switch.", "nodes": [ { "id": "rank0", "label": "GPU 0 rank 0", "kind": "host", "x": 0, "y": 60, "detail": "NVIDIA RTX 4000 Ada Generation, 20475 MiB, PCI address 0000:00:02. NCCL calls the process driving it rank 0." }, { "id": "phb", "label": "PCIe host bridge", "kind": "net", "x": 380, "y": 60, "detail": "The bridge inside the CPU. nvidia-smi topo -m labels this link PHB. NCCL rates the same path at 24.0 GB/s in its own topology dump, where it prints GPU/0-30 (4/24.0/PHB). The measured all-reduce reaches 22.08 GB/s, so the estimate is honest." }, { "id": "rank1", "label": "GPU 1 rank 1", "kind": "host", "x": 760, "y": 60, "detail": "The second card, rank 1, PCI address 0000:00:03. Same model, same 20475 MiB." }, { "id": "hostmem", "label": "Host memory (SHM)", "kind": "sensor", "x": 380, "y": 260, "detail": "Ordinary system RAM. The fallback route copies every buffer out of GPU memory into shared host memory, then into the other card. Two extra hops, and the measured all-reduce drops from 22.08 GB/s to 17.95 GB/s at 256 MB." } ], "edges": [ { "from": "rank0", "to": "phb", "label": "P2P/direct", "kind": "inband" }, { "from": "phb", "to": "rank1", "kind": "inband" }, { "from": "rank0", "to": "hostmem", "label": "SHM/direct", "kind": "mgmt" }, { "from": "hostmem", "to": "rank1", "kind": "mgmt" } ], "toggle": { "label": "Transport:", "on": "P2P enabled (default)", "off": "NCCL_P2P_DISABLE=1", "dimOn": ["hostmem", "e:rank0-hostmem", "e:hostmem-rank1"], "dimOff": ["e:rank0-phb", "e:phb-rank1"] } }
Flip the switch and watch which route lights up. Notice what never changes: the bridge. Both routes cross it. The difference is that the fast route hands bytes straight from one card to the other, and the slow route parks them in host memory on the way. That direct hand-off is called P2P, short for peer to peer.
nvidia-smi told you what the hardware permits. It says nothing about what your job actually gets. For that you need a benchmark that performs a real all-reduce and times it.
nccl-tests is that benchmark, an official NVIDIA suite, and all_reduce_perf is the tool inside it that runs the exact operation your training job runs. Four flags do the work here:
-b 8 is the smallest message to test, 8 bytes.-e 256M is the largest, 256 megabytes.-f 2 doubles the size at each step, so the sweep is 8, 16, 32, and so on.-g 2 uses 2 GPUs.A real training step moves tens or hundreds of megabytes, so the far end of that sweep is the part that pays your bills.
prompt: ops@gpu-node-07:~$ answer: all_reduce_perf -b 8 -e 256M -f 2 -g 2 output:
#
#
8 2 float sum -1 8.41 0.00 0.00 0 8.08 0.00 0.00 0 16 4 float sum -1 8.13 0.00 0.00 0 8.12 0.00 0.00 0 32 8 float sum -1 8.08 0.00 0.00 0 8.44 0.00 0.00 0 64 16 float sum -1 9.94 0.01 0.01 0 8.14 0.01 0.01 0 128 32 float sum -1 8.29 0.02 0.02 0 8.01 0.02 0.02 0 256 64 float sum -1 8.13 0.03 0.03 0 8.23 0.03 0.03 0 512 128 float sum -1 8.08 0.06 0.06 0 8.05 0.06 0.06 0 1024 256 float sum -1 8.18 0.13 0.13 0 8.11 0.13 0.13 0 2048 512 float sum -1 8.70 0.24 0.24 0 8.10 0.25 0.25 0 4096 1024 float sum -1 9.18 0.45 0.45 0 8.78 0.47 0.47 0 8192 2048 float sum -1 10.08 0.81 0.81 0 9.90 0.83 0.83 0 16384 4096 float sum -1 10.84 1.51 1.51 0 10.71 1.53 1.53 0 32768 8192 float sum -1 12.32 2.66 2.66 0 12.24 2.68 2.68 0 65536 16384 float sum -1 15.93 4.11 4.11 0 15.73 4.17 4.17 0 131072 32768 float sum -1 23.26 5.64 5.64 0 23.24 5.64 5.64 0 262144 65536 float sum -1 38.85 6.75 6.75 0 38.78 6.76 6.76 0 524288 131072 float sum -1 59.72 8.78 8.78 0 59.22 8.85 8.85 0 1048576 262144 float sum -1 95.71 10.96 10.96 0 95.49 10.98 10.98 0 2097152 524288 float sum -1 159.14 13.18 13.18 0 158.05 13.27 13.27 0 4194304 1048576 float sum -1 282.76 14.83 14.83 0 282.64 14.84 14.84 0 8388608 2097152 float sum -1 532.46 15.75 15.75 0 531.62 15.78 15.78 0 16777216 4194304 float sum -1 1030.44 16.28 16.28 0 1030.44 16.28 16.28 0 33554432 8388608 float sum -1 2034.87 16.49 16.49 0 2031.91 16.51 16.51 0 67108864 16777216 float sum -1 4006.09 16.75 16.75 0 3980.06 16.86 16.86 0 134217728 33554432 float sum -1 7735.91 17.35 17.35 0 7737.74 17.35 17.35 0 268435456 67108864 float sum -1 14956.0 17.95 17.95 0 15043.9 17.84 17.84 0
#
# hint: Type it exactly: all_reduce_perf -b 8 -e 256M -f 2 -g 2
Read the columns you will actually use. size is the message in bytes. time is how long one all-reduce took, in us, meaning microseconds, millionths of a second. #wrong is 0 on every row, which is the benchmark confirming the math came out correct. Nothing here is broken.
Then the two bandwidth columns. algbw is algorithm bandwidth: message size divided by time, the rate your application sees. busbw is bus bandwidth: what the link underneath had to carry to make that happen. For an all-reduce across two ranks the two numbers are identical, which is why every row above shows them matching. On bigger jobs they diverge, and busbw is the one to compare against hardware.
The bottom row is the one that matters: 256 MB moved at 17.95 GB/s. The run averaged 6.57446 across the whole sweep. Hold on to both numbers.
A benchmark number alone means nothing. 17.95 GB/s is neither good nor bad until you compare it to this same machine on a day it behaved.
That is why teams save baselines. The identical sweep was captured on this node last week, when the nightly job still finished on time. Here are the two runs side by side, same hardware, same command, seven days apart:
There is the incident, in one table. The tiny messages are untouched. The large ones lost about a fifth of their throughput, and the large ones are what a training step actually sends.
>>> The route changed. An 8 byte all-reduce spends almost all of its 8 microseconds on fixed startup cost, so the path barely matters. A 256 MB all-reduce spends its time actually pushing bytes, so the path is the whole story. Throttling would drag everything down together. A cliff that only appears under load is the signature of a slower route.
You know the operation got slower. You know the hardware is healthy. You know small transfers are fine. Nothing in any log file has said a word.
Commit to an answer before you read on.
>>> Ask NCCL which route it picked. NCCL decides how each pair of ranks will talk at startup, and it can be made to say so out loud. Temperature and clocks were already clean. Rerunning tells you the slowdown is real, which you knew. Only the route explains it.
NCCL is silent by default. Set the environment variable NCCL_DEBUG=INFO in front of the command and it narrates its entire startup. You get the plugins it found, the network it chose, how it built its rings, and one line per connection naming the transport it picked. A transport is simply the mechanism it uses to move bytes between two ranks.
That output is hundreds of lines. You want the handful that name a route, and every one of them contains the word via, so pipe the run through grep via. Use -b 256M -e 256M here, a single message size, because the transport choice is made once at startup and you do not need the whole sweep to read it.
prompt: ops@gpu-node-07:~$ answer: NCCL_DEBUG=INFO all_reduce_perf -b 256M -e 256M -g 2 2>&1 | grep via ||| NCCL_DEBUG=INFO all_reduce_perf -b 256M -e 256M -g 2 | grep via output: localhost:24162:24189 [1] NCCL INFO Channel 00 : 1[1] -> 0[0] via SHM/direct localhost:24162:24190 [0] NCCL INFO Channel 00 : 0[0] -> 1[1] via SHM/direct localhost:24162:24189 [1] NCCL INFO Channel 01 : 1[1] -> 0[0] via SHM/direct localhost:24162:24190 [0] NCCL INFO Channel 01 : 0[0] -> 1[1] via SHM/direct localhost:24162:24189 [1] NCCL INFO Channel 02 : 1[1] -> 0[0] via SHM/direct localhost:24162:24190 [0] NCCL INFO Channel 02 : 0[0] -> 1[1] via SHM/direct localhost:24162:24189 [1] NCCL INFO Channel 03 : 1[1] -> 0[0] via SHM/direct localhost:24162:24190 [0] NCCL INFO Channel 03 : 0[0] -> 1[1] via SHM/direct hint: Put NCCL_DEBUG=INFO in front of the command, then pipe it into grep via
Read one line, left to right. localhost is the node name. 24162 is the process id and 24189 the thread id, so yours will show different numbers. [1] is the rank speaking. Channel 01 is one of four parallel lanes NCCL built between the cards. 1[1] -> 0[0] is the direction, rank 1 sending to rank 0.
Then the payload of the whole lesson: via SHM/direct. SHM is shared memory, ordinary system RAM. Every byte those two GPUs exchange is being copied out to host memory and back in. That is the slow route from the diagram, and NCCL chose it without complaining once.
Here is the same grep from last week's baseline capture, on this same node, before anything changed:
localhost:24133:24160 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/direct pointer
localhost:24133:24161 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/direct pointer
localhost:24133:24160 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/direct pointer
localhost:24133:24161 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/direct pointer
localhost:24133:24160 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/direct pointer
localhost:24133:24161 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/direct pointer
localhost:24133:24160 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/direct pointer
localhost:24133:24161 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/direct pointerEight lines then, eight lines now, same channels, same directions. One phrase differs. P2P/direct pointer became SHM/direct, and that is the entire incident. Card to card became card to RAM to card.
Learn the healthy string by sight, because you cannot notice a downgrade you have never seen. NCCL has a ladder of transports and silently walks down it whenever a rung is unavailable:
{ "height": 400, "caption": "NCCL picks the highest rung available, silently. Click each rung for its measured cost on this node.", "nodes": [ { "id": "pick", "label": "NCCL picks a transport", "kind": "bmc", "x": 0, "y": 170, "detail": "Chosen once, at startup, per pair of ranks. It is announced only when NCCL_DEBUG=INFO is set, and never repeated later." }, { "id": "p2p", "label": "via P2P/direct pointer", "kind": "host", "x": 460, "y": 20, "detail": "Card writes straight into the other card across PCIe. 22.08 GB/s at 256 MB, 8.49873 average across the sweep, four channels. This is healthy on this node." }, { "id": "shm", "label": "via SHM/direct", "kind": "sensor", "x": 460, "y": 170, "detail": "Bytes detour through shared host memory. 17.95 GB/s at 256 MB, 6.57446 average, still four channels. This is what NCCL_P2P_DISABLE=1 buys you." }, { "id": "net", "label": "[send] via NET/Socket/0", "kind": "nic", "x": 460, "y": 320, "detail": "Bytes go out through the network stack, even between two cards in one box. Measured to 64 MB: 2.73 GB/s there, where the direct path does 21.75 GB/s at the same size. Average 0.86666. Only two channels survive, and each line gains a [send] or [receive] tag." } ], "edges": [ { "from": "pick", "to": "p2p", "label": "first choice", "kind": "inband" }, { "from": "pick", "to": "shm", "
Practice The All-Reduce That Got Slower in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.