Learn › GPU Operations & NCCL Troubleshooting › NCCL Troubleshooting
Diagnose NCCL bootstrap failures, the class of error where a job dies before any collective runs. Read the warning, clear the hardware, find the interface NCCL chose from its own log, and pin NCCL_SOCKET_IFNAME correctly on a real two-GPU node.
It is 02:14 at the Sea-West facility. A training job that ran clean yesterday afternoon is failing on ml-node-07, a two-GPU machine in Row 4.
It is not failing slowly. Not after an hour, not halfway through the first pass over the data. It dies about three seconds after launch, every single time.
The on-call engineer has already ruled out the obvious. Both GPUs are present. The driver is loaded. Nothing is overheating, and nothing has crashed.
Before you read another word, run the job yourself and watch it die.
This is a Practice Zone. The boxes below are a safe sandbox that checks only the one command being taught. The real ml-node-07 comes at the end, under Ready to practice.
prompt: engineer@ml-node-07:~$ answer: NCCL_SOCKET_IFNAME=doesnotexist0 all_reduce_perf -b 8M -e 8M -g 2 output:
#
[2026-07-29 07:21:55] localhost:24281:24281 [0] bootstrap.cc:126 (bootstrapNetInit) NCCL WARN Bootstrap : no socket interface found localhost: Test NCCL failure common.cu:1550 'invalid usage (run with NCCL_DEBUG=WARN for details) / ' .. localhost pid 24281: Test failure common.cu:1454 hint: The variable is NCCL_SOCKET_IFNAME set to doesnotexist0, placed in front of all_reduce_perf -b 8M -e 8M -g 2
Now read what just happened, slowly. Both GPUs are found. They are printed by name: NVIDIA RTX 4000 Ada Generation, one at PCI address 0000:00:02, one at 0000:00:03.
Then the job dies. And the warning that kills it says nothing about a GPU.
It says socket. It says Bootstrap. Those are networking words, on a machine where both GPUs are bolted into the same box, inches apart.
That is the mystery this lesson closes.
NCCL (the NVIDIA Collective Communications Library, pronounced "nickel") is the code that moves data between GPUs during training. When thousands of GPUs average their results together, NCCL is doing it.
A collective is one of those group operations. The common one is an all-reduce: every GPU contributes a number, and every GPU gets the total back. A rank is simply one participant, one GPU's seat at the table. This job has 2 ranks, because it has 2 GPUs.
Here is the part almost nobody is taught. NCCL starts in two separate phases, and they use completely different machinery.
Phase 1 is bootstrap. The ranks have to find each other before they can do anything. They exchange addresses over an ordinary TCP socket, the same kind of connection a web browser makes. To open a socket, the operating system needs a network interface: a named connection point like eth0. Nothing in this phase touches a GPU.
Phase 2 is transport. Only after every rank has been found does NCCL choose how to actually move the tensor data, picking the fastest path available between the GPUs.
Your job died in phase 1. It never reached phase 2. That is why the error is a networking error on a single machine.
{ "height": 460, "caption": "NCCL starts twice. Click every box, then flip the interface lookup and watch how far the job gets.", "nodes": [ { "id": "job", "label": "training job", "kind": "admin", "x": 0, "y": 180, "detail": "The job on ml-node-07. It calls into NCCL before it touches a single GPU, which is why a broken network setting can kill it in three seconds." }, { "id": "boot", "label": "Phase 1: bootstrap", "kind": "net", "x": 260, "y": 60, "detail": "Ranks find each other. This runs over a plain TCP socket and needs a named network interface. It happens even when every rank is on the same machine." }, { "id": "iface", "label": "network interface", "kind": "nic", "x": 560, "y": 60, "detail": "The interface NCCL binds its bootstrap socket to. On the healthy run of this exact node the log named it: NET/Socket : Using [0]eth0:172.232.181.15<0>" }, { "id": "fail", "label": "no socket interface found", "kind": "sensor", "x": 860, "y": 60, "detail": "When the name matches nothing, bootstrap.cc:126 raises NCCL WARN Bootstrap : no socket interface found, the harness reports invalid usage, and the process exits 3." }, { "id": "transport", "label": "Phase 2: transport", "kind": "net", "x": 260, "y": 320, "detail": "Only reached if phase 1 succeeded. Here NCCL picks how to move tensors: direct GPU to GPU, through shared memory, or over the network." }, { "id": "gpus", "label": "GPU0 + GPU1", "kind": "host", "x": 560, "y": 320, "detail": "Two RTX 4000 Ada, 20475 MiB each. nvidia-smi topo -m reports PHB, a PCIe host bridge, and the P2P matrix reports OK. There is no NVLink on this hardware." } ], "edges": [ { "from": "job", "to": "boot", "label": "starts here", "kind": "mgmt" }, { "from": "boot", "to": "iface", "label": "bind a TCP socket", "kind": "mgmt" }, { "from": "iface", "to": "fail", "label": "name matches nothing", "kind": "plain" }, { "from": "boot", "to": "transport", "label": "only if phase 1 succeeds", "kind": "inband" }, { "from": "transport", "to": "gpus", "label": "P2P over PCIe", "kind": "inband" } ], "toggle": { "label": "Interface lookup:", "on": "Match found (eth0)", "off": "No match (doesnotexist0)", "dimOn": ["fail", "e:iface-fail"], "dimOff": ["transport", "gpus", "e:boot-transport", "e:transport-gpus"] } }
Flip the lookup to the broken setting. Notice what goes dark: the entire bottom row. The GPUs are never reached, never asked to do anything, never given the chance to fail. They are bystanders.
NCCL was built by NVIDIA and released in 2015. The first version solved one machine only, moving data between GPUs inside a single chassis.
The next version had to work across many machines, so the ranks needed a way to find each other over a network. Rather than keep two separate startup paths, NVIDIA kept one. Every job now begins with the same socket rendezvous, whether the ranks are in one box or in four hundred.
That single design decision is the whole reason a two-GPU machine can be taken down by a bad interface name.
This is a common shape in systems engineering. A general mechanism gets used for the simple case too, so the simple case inherits the general case's failure modes. Your two-GPU node fails like a cluster because it starts up like a cluster.
>>> Bootstrap always uses a socket. One startup path serves both the single-node and the multi-node case, so the socket step never gets skipped. Note the third answer is a real trap: this hardware genuinely has no NVLink, which makes the wrong reason sound convincing. Bootstrap would still need an interface even if NVLink were present.
Good engineers do not take a warning message on trust. Before accepting that the hardware is innocent, prove it.
nvidia-smi is the driver's command line tool. Its topo -m form prints the topology matrix: how every GPU in the machine is wired to every other GPU.
Before you run it, commit to an answer. If these two GPUs cannot reach each other directly, that would explain a failure. Do you expect this matrix to show a healthy link, or a broken one?
prompt: engineer@ml-node-07:~$ answer: nvidia-smi topo -m output: GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PHB 0-7 0 N/A GPU1 PHB X 0-7 0 N/A
Legend:
X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks hint: The driver tool is nvidia-smi, and the topology form is topo with the -m flag.
Read the matrix like a mileage chart. Find the row for GPU0, follow it to the GPU1 column, and the cell says PHB.
The legend decodes it: PHB means the two GPUs talk through a PCIe host bridge, the connection point on the CPU. Their traffic goes GPU0 into the CPU, then back out to GPU1.
Notice what is absent. The legend lists NV# for NVLink, NVIDIA's dedicated high speed GPU to GPU cable. No cell in this matrix says NV. This hardware has no NVLink, and that is normal. The RTX 4000 Ada is a budget accelerator, and PCIe is the layout most engineers actually meet.
The same tool also reports whether the GPUs can address each other directly, which NVIDIA calls P2P (peer to peer). Here is that report from this node, for reading rather than typing:
GPU0 GPU1
GPU0 X OK
GPU1 OK X
Legend:
X = Self
OK = Status Ok
CNS = Chipset not supported
GNS = GPU not supported
TNS = Topology not supported
NS = Not supported
DR = Disabled by regkey
U = UnknownOK in both directions. Hold both facts together, because together they are the mental model for a budget two-GPU node: no NVLink, but peer to peer still works over PCIe. The hardware is fine. It was always fine.
Every program hands back a number when it finishes, called the exit code. Zero means success. Anything else means a specific kind of failure, and the shell keeps the most recent one in $?.
Reading it costs nothing and is the fastest way to tell a crash apart from a refusal. Ask the shell what the dead job returned.
prompt: engineer@ml-node-07:~$ answer: echo $? output: 3 hint: The echo command prints things, and the shell stores the last exit code in the variable $?
Three, not zero. Now pair that number with the words the harness printed: invalid usage.
This matters more than it looks. The job did not segfault. It did not run out of memory. It did not time out waiting for a peer. It was told to do something impossible and refused immediately.
invalid usage is NCCL saying: you asked for a thing that does not exist. That is a configuration failure, not a hardware failure. Configuration failures are the ones you can fix in seconds, once you know where to look.
There is a second clue in that failed run, and it is easy to walk past. Look again at the order things were printed. The two GPUs appeared first, by name. The warning came after.
>>> The harness enumerated them first. Those device lines come from the test program itself, before it hands control to NCCL. So the GPU list proves the driver works and the cards are visible. It says nothing about NCCL, which had not run yet.
You know the job dies at bootstrap. You know the hardware is fine. The missing piece is what a working bootstrap looks like, so you have something to compare against.
NCCL will tell you, if you ask. Setting NCCL_DEBUG=INFO turns on its startup log, which reports every decision it makes while starting.
That log is hundreds of lines, so filter it. grep prints only the lines containing the text you name. The line worth finding is the one where NCCL announces its socket choice, tagged NET/Socket.
prompt: engineer@ml-node-07:~$ answer: NCCL_DEBUG=INFO all_reduce_perf -b 128M -e 128M -g 2 2>&1 | grep 'NET/Socket' output: localhost:24133:24153 [0] NCCL INFO NET/Socket : Using [0]eth0:172.232.181.15<0> hint: Set NCCL_DEBUG to INFO in front of the same test command, then send both output streams into grep and search for the text NET/Socket
One line, and it is the whole answer. NCCL found an interface named eth0, carrying the address 172.232.181.15, and bound its bootstrap socket to it.
Now put the two runs side by side. This is the entire diagnostic skill of this lesson:
Same machine. Same GPUs. Same command. The only difference is whether the name you handed NCCL matched a real interface.
One of those lines names an interface. The other reports finding none. When a job dies in the first three seconds, that single line is what you go looking for.
The 2>&1 in that command matters. NCCL writes its log to the error stream, not the normal one, so without redirecting it the grep would find nothing. Missing that redirect is why engineers often swear the debug output is not working.
Stop and take stock, because you just crossed the line that separates guessing from diagnosing. You can now look at a job that died in three seconds and say which of its two startup phases failed, and prove the hardware innocent while you do it. That is the skill.
Here is the rule that governs everything you just saw. When NCCL needs a bootstrap interface, it does one of two things.
If NCCL_SOCKET_IFNAME is not set, NCCL scans the machine and picks one itself. It skips the loopback interface lo, because loopback cannot reach another machine. It skips interfaces named docker* for the same reason. On ml-node-07 that scan landed on eth0, exactly as the log reported.
If NCCL_SOCKET_IFNAME is set, NCCL stops guessing and obeys you. It matches the names you gave, and if none of them exist, it does not quietly fall back. It fails, which is what you saw at 02:14.
The variable accepts more than one plain name:
So why would anyone set it at all? Because on real machines the automatic scan can pick wrong.
A multi-homed host is one with several network interfaces. A GPU node commonly has a slow management interface and a fast data interface. The scan may bind to the slow one, and your training job crawls with nothing obviously broken.
Containers make it worse. Inside a container the interface is often named something unfamiliar, and the name that worked on the host means nothing inside. That is how a setting copied from a working machine becomes a dead job on another.
The dangerous version of this failure is not the crash. A crash is honest and takes seconds to read. The quiet version is NCCL binding to the wrong interface and succeeding, leaving you with a job that trains at a fraction of the speed it should.
>>> The interface name does not exist inside the container. Container networking usually renames or replaces interfaces, so a value carried over from the host matches nothing. If the container could not see the GPUs, the failure would come earlier and would name devices, not sockets.
No commands are shown from here on. You have used every piece already. Recall them.
prompt: engineer@ml-node-07:~$ answer: NCCL_DEBUG=INFO all_reduce_perf -b 128M -e 128M -g 2 2>&1 | grep 'NET/IB' output: localhost:24133:24153 [0] NCCL INFO NET/IB : No device found. localhost:24133:24153 [0] NCCL INFO NET/IB : Using [RO]; OOB eth0:172.232.181.15<0> hint: Same debug run and same filter you used for the socket line. Only the search text changes: NCCL tags its InfiniBand probe NET/IB
InfiniBand is a high speed network fabric used in large clusters. No device found settles it: this node has none, so NCCL falls back to ordinary sockets. That is the correct answer to give the vendor.
The ticket needs the exact warning, not a paragraph of surrounding noise. Break the job again, and this time show only the line that matters.
prompt: engineer@ml-node-07:~$ answer: NCCL_SOCKET_IFNAME=doesnotexist0 all_reduce_perf -b 8M -e 8M -g 2 2>&1 | grep 'NCCL WARN' output: [2026-07-29 07:21:55] localhost:24281:24281 [0] bootstrap.cc:126 (bootstrapNetInit) NCCL WARN Bootstrap : no socket interface found hint: The same broken run from the cold open, with both streams piped into grep. NCCL tags its serious complaints with the words NCCL WARN
Isolated, the line gives up more than it did in the noise. bootstrap.cc:126 is the exact source file and line inside NCCL that gave up, and bootstrapNetInit is the function name. When you file a ticket, that is what you paste.
Last one. Phase 2 announces the transport it chose, once per channel. Go and read that announcement.
prompt: engineer@ml-node-07:~$ answer: NCCL_DEBUG=INFO all_reduce_perf -b 128M -e 128M -g 2 2>&1 | grep 'via P2P' output: localhost:24133:24160 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/direct pointer localhost:24133:24161 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/direct pointer localhost:24133:24160 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/direct pointer localhost:24133:24161 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/direct pointer localhost:24133:24160 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/direct pointer localhost:24133:24161 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/direct pointer localhost:24133:24160 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/direct pointer localhost:24133:24161 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/direct pointer hint: Same debug run and same filter habit. Phase 2 announces its chosen transport on every channel with the words via P2P
Eight lines, four channels, both directions. via P2P/direct pointer means each GPU writes straight into the other's memory over PCIe. The network carried the introductions in phase 1 and then stepped aside.
You opened this lesson unable to explain a three second death. You can now name the phase, prove the hardware innocent, and fix the setting.
Practice The Job That Never Starts in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.