Learn › GPU Operations & NCCL Troubleshooting › NCCL Fundamentals
nvidia-smi topo -m - a hands-on Linux lab on a real virtual machine.
Two GPUs in one machine can be neighbors or strangers. Read nvidia-smi topo -m and the P2P capability matrix on real hardware, decode PHB, PIX, PXB, NODE, SYS and NV, and learn why this box has no NVLink and what that costs.
It is 14:40 on a Thursday. Your training job has run on one GPU for three weeks. This morning the platform team moved it to a bigger machine and closed the ticket with six words: we doubled your GPUs, enjoy.
You log in. There really are two cards. The job runs. It is not twice as fast.
Nobody lied to you. You did get a second GPU. What nobody mentioned is the part that decides everything: what sits between the two cards.
A GPU is a graphics processing unit, the accelerator card that does the heavy arithmetic in machine learning. Two of them in one machine can be next-door neighbors. They can also be strangers who pass every message through a third party.
One command tells you which. Three letters in its output set the speed ceiling for every job this machine will ever run.
The machine waiting in the bay below is that box: two NVIDIA RTX 4000 Ada Generation cards, 20475 MiB of memory each. By the end of this lesson you will know exactly what you got.
This is a Practice Zone. The black boxes below are a safe sandbox that checks one command at a time, and the diagrams react to you: click the boxes, flip the switches. The real machine comes at the end, under Ready to practice.
Every NVIDIA card is run by a driver, the software that lets the operating system talk to the hardware. Installed alongside it is a command called nvidia-smi, short for NVIDIA System Management Interface.
nvidia-smi is the window into every NVIDIA GPU on the machine. Most people know it for the big status table. It has a second job almost nobody uses, and that job is this lesson.
The subcommand is topo, short for topology: the shape of the connections, who is wired to whom. Add -m for matrix and the driver prints a grid of every GPU against every other GPU, with the connection type in each cell.
Here is why the subcommand exists at all. When a machine held one accelerator, nobody had to ask how it was wired. Once a single chassis started holding four or eight, every performance question began with the same sentence: which card is next to which.
NVIDIA answered that with two things. NVLink, a dedicated high speed link that runs directly between GPUs, arrived with the Pascal generation in 2016. Everything without NVLink falls back to PCIe, the general purpose expansion bus that also carries your disks and your network card. topo -m is the driver telling you, per pair, which road your bytes are on.
Take a position first. Guessing and being wrong is what makes the real answer stick.
>>> It depends on the machine. That is the uncomfortable part. Two boxes with identical spec sheets, identical card models and identical prices can be wired completely differently inside, and the difference is invisible until you ask the driver. This is why engineers who run GPU fleets check the wiring before they trust a benchmark.
Now find out. Type this exact command in the practice terminal and press Enter:
nvidia-smi topo -m
prompt: ops@gpu-fabric:~$ answer: nvidia-smi topo -m output: GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PHB 0-7 0 N/A GPU1 PHB X 0-7 0 N/A
Legend:
X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks hint: Three words, in order: the tool, the subcommand topo, and the flag -m.
Ignore the legend for a moment. The entire answer is in the first three lines.
It is a grid with GPU0 and GPU1 on both axes. Read a cell as "how do these two connect". The diagonal cells say X , which the legend defines as Self: a card compared with itself. Those two cells carry no information.
The other two cells are the ones you came for, and both say the same thing: PHB.
Now the delta that teaches this lesson. The legend lists seven possible values a cell can hold. Six of them describe PCIe paths of different lengths. The seventh, NV#, is NVLink. Your matrix does not contain it. Not once, in either direction.
That absence is the finding. This machine has no NVLink at all, and no amount of tuning will conjure any. Every byte GPU0 sends to GPU1 goes out over PCIe, up to the host bridge on the CPU, and back down the other side.
Two more columns come free with the same output. CPU Affinity reads 0-7, meaning both cards are closest to CPU cores 0 through 7. NUMA Affinity reads 0, meaning both hang off the same memory region of the machine. Same cores, same memory neighborhood, for both cards.
{ "height": 400, "caption": "The road a byte takes from GPU0 to GPU1 on this machine. Click any box, then flip the machine switch.", "nodes": [ { "id": "gpu0", "label": "GPU0 RTX 4000 Ada", "kind": "host", "x": 0, "y": 30, "detail": "NVIDIA RTX 4000 Ada Generation, 20475 MiB. NCCL sees it as device 0 at PCI address 0000:00:02." }, { "id": "gpu1", "label": "GPU1 RTX 4000 Ada", "kind": "host", "x": 0, "y": 300, "detail": "NVIDIA RTX 4000 Ada Generation, 20475 MiB. NCCL sees it as device 1 at PCI address 0000:00:03." }, { "id": "slot0", "label": "PCIe 0000:00:02", "kind": "nic", "x": 300, "y": 30, "detail": "The PCIe slot GPU0 sits in. PCIe is the general purpose expansion bus, shared with disks and network cards." }, { "id": "slot1", "label": "PCIe 0000:00:03", "kind": "nic", "x": 300, "y": 300, "detail": "The PCIe slot GPU1 sits in. Different slot, same bus, same host bridge above it." }, { "id": "nvbridge", "label": "NVLink (absent here)", "kind": "psu", "x": 300, "y": 165, "detail": "A direct GPU-to-GPU link. This machine has none: NV# never appears in the matrix. Flip the switch to see the shape it would have." }, { "id": "phb", "label": "PCIe host bridge (CPU)", "kind": "bmc", "x": 640, "y": 165, "detail": "The turning point. PHB means traffic climbs to this bridge, typically part of the CPU, and comes back down the other side. Both cards report CPU affinity 0-7 and NUMA affinity 0, so this is one bridge, not two." } ], "edges": [ { "from": "gpu0", "to": "slot0", "kind": "mgmt" }, { "from": "slot0", "to": "phb", "label": "PHB", "kind": "mgmt" }, { "from": "gpu1", "to": "slot1", "kind": "mgmt" }, { "from": "slot1", "to": "phb", "label": "PHB", "kind": "mgmt" }, { "from": "gpu0", "to": "nvbridge", "label": "NV#", "kind": "power" }, { "from": "gpu1", "to": "nvbridge", "label": "NV#", "kind": "power" } ], "toggle": { "label": "Machine:", "on": "This box: 2x RTX 4000 Ada (PHB)", "off": "A different box, with NVLink (NV#)", "dimOn": ["nvbridge", "e:gpu0-nvbridge", "e:gpu1-nvbridge"], "dimOff": ["slot0", "slot1", "phb", "e:gpu0-slot0", "e:slot0-phb", "e:gpu1-slot1", "e:slot1-phb"] } }
Click both GPUs, then flip the machine switch. Notice what does not change: the cards themselves. Same model, same memory, same driver. Only the road between them moves, and that road is the whole difference.
Most people skim the legend once and never read it again. That is a mistake, because the legend is not a glossary. It is a ladder, ordered by distance.
Read the values from closest to farthest and the pattern appears. Each rung adds one more thing the traffic has to cross:
A NUMA node is one CPU socket together with the memory attached to it. On a two socket server, crossing from one socket to the other is the longest trip inside the box, which is why SYS sits at the bottom.
NV# is the odd one out. It is not a longer or shorter PCIe path, it is a different road: NV2 means two bonded NVLinks, NV12 means twelve. When a matrix cell says NV#, PCIe is not involved in that hop at all.
{ "height": 320, "caption": "Every value a cell can hold, ordered by distance. Click a rung to read the definition the driver prints.", "nodes": [ { "id": "pix", "label": "PIX", "kind": "host", "x": 0, "y": 40, "detail": "PIX = Connection traversing at most a single PCIe bridge. The shortest PCIe hop two cards can have." }, { "id": "pxb", "label": "PXB", "kind": "host", "x": 190, "y": 40, "detail": "PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge). Still inside the switch fabric, never climbing to the CPU." }, { "id": "phb", "label": "PHB", "kind": "bmc", "x": 380, "y": 40, "detail": "PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU). This is what both off-diagonal cells of your matrix say." }, { "id": "node", "label": "NODE", "kind": "net", "x": 570, "y": 40, "detail": "NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node." }, { "id": "sys", "label": "SYS", "kind": "net", "x": 760, "y": 40, "detail": "SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI). Two CPU sockets apart, the longest trip inside one box." }, { "id": "nv", "label": "NV# (not on this machine)", "kind": "psu", "x": 380, "y": 210, "detail": "NV# = Connection traversing a bonded set of # NVLinks. Not a rung on this ladder: a separate GPU-to-GPU road that skips PCIe. Your matrix never prints it." } ], "edges": [ { "from": "pix", "to": "pxb", "label": "farther", "kind": "plain" }, { "from": "pxb", "to": "phb", "label": "farther", "kind": "plain" }, { "from": "phb", "to": "node", "label": "farther", "kind": "plain" }, { "from": "node", "to": "sys", "label": "farther", "kind": "plain" } ] }
Click PHB and then NV#. The rung you land on is not a detail for a spec sheet. It is the budget every collective operation on this machine has to live inside.
You now know this box has no NVLink and that both cells say PHB. Take a position on what that means before the driver answers.
>>> Yes. This is the single most misread thing about GPU topology. Distance and permission are two different questions. topo -m answers how far apart the cards are. Whether one card is allowed to address the other directly is a separate capability called peer-to-peer, usually written P2P, and the driver reports it separately. NVLink makes P2P faster. It is not what makes P2P possible.
The same subcommand answers it, with a different flag. -p2p asks about peer-to-peer capability, and it takes one letter naming which capability you want. r means read.
Type this exact command:
nvidia-smi topo -p2p r
prompt: ops@gpu-fabric:~$ answer: nvidia-smi topo -p2p r output: GPU0 GPU1 GPU0 X OK GPU1 OK X
Legend:
X = Self OK = Status Ok CNS = Chipset not supported GNS = GPU not supported TNS = Topology not supported NS = Not supported DR = Disabled by regkey U = Unknown hint: Same tool and same topo subcommand as before. The flag is -p2p and it needs one letter after it: r for read.
OK in both off-diagonal cells. GPU0 can read GPU1 memory directly, and GPU1 can read GPU0 memory directly, across PCIe, with no copy staged in system RAM on the way.
Hold the two outputs side by side, because this pairing is the entire mental model for a budget two-card machine:
topo -m says PHB. The cards are far apart by GPU standards, all the way up to the CPU and back.topo -p2p r says OK. They are still allowed to address each other directly.Both are true. Neither cancels the other.
The rest of that legend is the failure list, worth knowing for the day one of them shows up. CNS means the chipset cannot do it. GNS means the card cannot. TNS means the wiring cannot. DR means somebody switched it off deliberately.
Stop and notice where you are. You just answered the question your team lead could not: not how many GPUs, but what kind of two-GPU machine this is. Model, count and price were on the invoice. The wiring was not, and the wiring is what you will feel.
Last position to take, and this one is about money.
>>> It sets a ceiling. Nothing errors, nothing warns, and small transfers look perfectly healthy. The rung shows up when the messages get big, which is exactly when a training job starts synchronizing real gradients. The next step shows the receipt from this very machine.
Two pieces of evidence, both captured on the same hardware you just inspected. You do not run these here. This is the preview of what the rest of this module measures.
First, NCCL. NCCL is the NVIDIA Collective Communications Library, the software that moves data between GPUs during training. Before it moves a single byte it builds its own map of the machine, and with debug logging on it prints that map:
localhost:24133:24153 [0] NCCL INFO GPU/0-20 :GPU/0-20 (0/5000.0/LOC) GPU/0-30 (4/24.0/PHB) CPU/0-ffffffffffffff (2/24.0/PHB) NET/0-0 (4/1.2/PHB) DEV/0-20 (1/5000.0/LOC) DEV/0-30 (3/24.0/PHB)Read the middle group: GPU/0-30 (4/24.0/PHB). That is NCCL describing the path from this GPU to the other one. Same three letters you read in the matrix, PHB, and beside them NCCL own estimate of the road capacity: 24.0 GB/s.
Second, the measurement. Running an AllReduce, the collective that averages gradients across GPUs, across message sizes from 8 bytes up to 256 MB prints 26 rows. Three of them carry the story. The ... marks rows cut to fit, not a gap in the run:
# size count type redop root time algbw busbw #wrong time algbw busbw #wrong
# (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)
...
8388608 2097152 float sum -1 397.03 21.13 21.13 0 397.40 21.11 21.11 0
...
67108864 16777216 float sum -1 3085.89 21.75 21.75 0 3083.85 21.76 21.76 0
...
268435456 67108864 float sum -1 12157.1 22.08 22.08 0 12137.1 22.12 22.12 0The busbw column is bus bandwidth, how fast the road between the cards is actually running. At a 256 MB message it reaches 22.08 GB/s against a modelled 24.0. The map matched the machine.
That number is your ceiling, and it is set by a rung on a ladder you can read in one command. For contrast, when this same box is forced off the direct PCIe path, the same 256 MB AllReduce drops to 17.95 GB/s. Nothing breaks. It just gets slower, quietly, forever.
The full output is right for a first look and wrong for a ticket. The grid comes first: one header line, then one row per GPU. On this two-card machine that is three lines. On an eight-card machine it would be nine. Everything after the grid is the legend.
So the form engineers actually use trims it. Type this exact command:
nvidia-smi topo -m | head -3
prompt: ops@gpu-fabric:~$ answer: nvidia-smi topo -m | head -3 output: GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PHB 0-7 0 N/A GPU1 PHB X 0-7 0 N/A hint: Print the matrix, then send it through a pipe into head, asking for the first 3 lines.
Three lines. That fits in a chat message, a ticket comment or a runbook, and it carries the whole answer.
Scaffolding off. No command shown from here.
A colleague asks a narrow question in chat: what does GPU0 connect to and how. Give them GPU0 row on its own. No header, no legend, no second row. One line out.
prompt: ops@gpu-fabric:~$ answer: nvidia-smi topo -m | grep ^GPU0 output: GPU0 X PHB 0-7 0 N/A hint: Two things you already know, joined by a pipe: print the matrix, then filter it with the pattern tool. Anchor the pattern to the start of the line, because the header row begins with a tab and would otherwise match too.
Same idea, other question. The runbook needs the peer-to-peer read capability as a bare grid, with the legend stripped off. Three lines.
prompt: ops@gpu-fabric:~$ answer: nvidia-smi topo -p2p r | head -3 output: GPU0 GPU1 GPU0 X OK GPU1 OK X hint: The capability question you asked earlier, sent through the same trimming pipe you used on the wiring matrix.
You started this lesson unable to say what kind of two-GPU machine you had been handed. You can now answer it in one line, from memory.
nvidia-smi topo -m # the wiring between every pair of GPUs
nvidia-smi topo -m | head -3 # just the grid, no legend
nvidia-smi topo -m | grep ^GPU0 # one GPU row, for a chat message
nvidia-smi topo -p2p r # whether the cards may address each otherAnd the readings that matter:
Time to do it on real hardware. The machine in the bay below is the same two-card box: two RTX 4000 Ada cards and a fabric you already know the shape of.
Five objectives close this ticket:
1. File the wiring matrix. The full output, legend and all, saved to ~/fabric-matrix.txt. 2. File the capability answer. The peer-to-peer read grid, saved to ~/fabric-p2p.txt. 3. Isolate one row. GPU0 row and nothing else, in ~/fabric-gpu0-row.txt
Practice Reading the Fabric in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.