Learn › GPU Operations & NCCL Troubleshooting › nvidia-smi Mastery
nvidia-smi --query-gpu=index,name,uuid,pci.bus_id --format=csv - a hands-on Linux lab on a real virtual machine.
Set CUDA_VISIBLE_DEVICES on a real two GPU machine and watch nvidia-smi ignore it completely, because nvidia-smi is an NVML tool and not a CUDA application. Then ask a CUDA process the same question, see the masked card renumbered to device 0, and learn why an out of range ordinal reports no CUDA-capable device is detected on perfectly healthy hardware.
It is 09:20 on a Tuesday. The ticket is one line: run the evaluation job on the second card only, the first one is reserved.
You know the switch for that. You set CUDA_VISIBLE_DEVICES=1, run nvidia-smi to confirm it took, and the terminal prints both cards. Index 0 and index 1, exactly as before.
So you assume the variable is wrong. Wrong name, wrong driver, wrong container flag. You spend twenty minutes there.
Then the job starts, prints one line naming the hardware it grabbed, and calls that card device 0.
Two tools, two answers, one machine, and neither of them is broken. By the end of this lesson you will know which question each tool is answering, and why the answer that looks wrong is the correct one.
The machine in the bay below is the box these outputs came from: two NVIDIA RTX 4000 Ada Generation cards, driver 610.43.02, Ubuntu 24.04.
This is a Practice Zone. The black boxes below are a safe sandbox that checks one command at a time, and the diagram reacts to you: click the boxes, flip the switch. The real machine comes at the end, under Ready to practice.
CUDA is the NVIDIA platform for running general purpose work on a GPU. Anything that trains a model or serves inference is a CUDA application.
An environment variable is a named value the shell hands to a program when that program starts. Write it in front of a command and it applies to that one run: NAME=value command.
CUDA_VISIBLE_DEVICES is the environment variable that decides which GPUs a CUDA application is allowed to see. Its value is a comma separated list of ordinals, and an ordinal is just a position in the list of cards the driver found. 0 is the first card, 1 is the second.
It is the most used GPU control in the industry. Every scheduler sets it. Every shared machine leans on it. And it has one behavior that catches nearly everybody the first time.
Never hide hardware you have not counted. Ask the driver for a list of the cards with three facts about each one.
--query-gpu= selects the fields you want, by name, separated by commas. --format=csv prints them as comma separated values, which is a plain table that both a person and a script can read.
Type this exact command in the practice terminal and press Enter:
nvidia-smi --query-gpu=index,name,uuid,pci.bus_id --format=csv
prompt: ops@gpu-mask:~$ answer: nvidia-smi --query-gpu=index,name,uuid,pci.bus_id --format=csv output: index, name, uuid, pci.bus_id 0, NVIDIA RTX 4000 Ada Generation, GPU-14c73d7e-6cf9-0056-e332-5a4081954c17, 00000000:00:02.0 1, NVIDIA RTX 4000 Ada Generation, GPU-da3db8be-0267-da92-25a8-86d6655b77c5, 00000000:00:03.0 hint: The tool name, then --query-gpu= with the four field names joined by commas, then --format=csv.
Every card in that output carries three identifiers, and they are not equally trustworthy.
index is a position in a list. It is the weakest of the three, because a list can be filtered and renumbered. That is the whole subject of this lesson.
uuid is the permanent name of the card, fixed at the factory. GPU-14c73d7e-6cf9-0056-e332-5a4081954c17 follows that card into another chassis, another machine, another year. Two cards can never share one.
pci.bus_id is the slot address on the PCI Express bus, the expansion bus that connects cards to the rest of the machine. Card 0 sits at 00000000:00:02.0 and card 1 at 00000000:00:03.0. Different slot, different address.
Note the shape of those addresses, because you are about to meet them again in a shorter form. nvidia-smi pads to 00000000:00:03.0 and CUDA programs usually print 0000:00:03. Same slot, same card.
The UUIDs above belong to the two cards these captures came from. Your machine prints its own, and a PCI address can differ from one machine to the next. The shape of the output is what carries over, not the exact strings.
Here is the fact this whole lesson turns on, and almost nobody is told it.
nvidia-smi is not a CUDA program. It is built on NVML, the NVIDIA Management Library, which is the management interface to the driver. NVML exists to inventory and administer hardware: temperature, power draw, memory, clocks, running processes.
A CUDA application is a different animal. It opens a context on a card, which is the working session through which it runs code and moves data.
CUDA_VISIBLE_DEVICES is read by the CUDA layer as it starts up. NVML never reads it, and never has. A management tool that hid cards because of an environment variable would be a broken management tool, because you would lose sight of the hardware you are paid to watch.
So the variable is not ignored by accident. It is ignored by design.
Take a position first. Being wrong here, on purpose, is the fastest way to make the real answer stick.
>>> Both cards, unchanged. nvidia-smi reports what the driver has, not what your next process will be allowed to touch. This single behavior is responsible for most of the time engineers lose on device masking, and it looks exactly like a broken setting. Run it and watch it happen.
Put the variable in front of the command so it applies to this one run only. Ask for the same fields as before, minus the PCI address, to keep the output narrow.
Type this exact command:
CUDA_VISIBLE_DEVICES=1 nvidia-smi --query-gpu=index,name,uuid --format=csv
prompt: ops@gpu-mask:~$ answer: CUDA_VISIBLE_DEVICES=1 nvidia-smi --query-gpu=index,name,uuid --format=csv output: index, name, uuid 0, NVIDIA RTX 4000 Ada Generation, GPU-14c73d7e-6cf9-0056-e332-5a4081954c17 1, NVIDIA RTX 4000 Ada Generation, GPU-da3db8be-0267-da92-25a8-86d6655b77c5 hint: Put CUDA_VISIBLE_DEVICES=1 in front of the same query command, and drop pci.bus_id from the field list.
Put the two UUIDs beside the ones from your first command. Same two cards, same order, same index numbers. The mask changed nothing at all.
This is the exact moment those twenty minutes get lost. The output looks like proof that the variable did not take, so people go hunting through spelling, driver versions and container runtime flags. All of it is wasted, because the variable did take. You asked the one tool on the machine that is not listening.
Say it plainly, because this is the sentence to carry out of the lesson: nvidia-smi can neither confirm nor deny a CUDA mask. Only a CUDA process can.
One more position to take, and this one is about numbering.
>>> Device 0. The mask does not hide a card and leave a hole where it was. It builds a fresh list of the survivors and numbers that list from zero. Mask to the second card and it becomes the only entry, so it becomes entry number zero. The next command proves it on real hardware.
all_reduce_perf is a benchmark from the NVIDIA nccl-tests suite, and unlike nvidia-smi it is a real CUDA application. Before it measures anything it prints which device it took, which is precisely the receipt this lesson needs.
Four flags keep the run tiny. -b 8 begins at 8 bytes, -e 8 ends at 8 bytes, -f 1 does not grow the size between runs, and -g 1 uses one GPU.
Type this exact command:
CUDA_VISIBLE_DEVICES=1 all_reduce_perf -b 8 -e 8 -f 1 -g 1
prompt: ops@gpu-mask:~$ answer: CUDA_VISIBLE_DEVICES=1 all_reduce_perf -b 8 -e 8 -f 1 -g 1 output:
#
#
8 2 float sum -1 3.06 0.00 0.00 0 0.12 0.07 0.00 0
#
# hint: Same variable in front, then the benchmark name, then the four flags: -b 8, -e 8, -f 1, -g 1.
One line in that output carries the entire lesson:
# Rank 0 Group 0 Pid 23978 on localhost device 0 [0000:00:03] NVIDIA RTX 4000 Ada GenerationRead it from the right. The card sits at PCI address 0000:00:03. Go back to your inventory: that address belongs to index 1, the second card. It is the card you asked for, so the mask worked perfectly.
Now read the middle of the line: device 0.
One card is visible and it is the right one, but CUDA calls it device 0. The visible list gets renumbered from zero, every time, without a word of warning.
That renumbering is the trap. An ordinal inside a masked process does not point at the same card as the same ordinal outside it. A log line reading "training on device 0" tells you nothing on its own about which physical card is warm.
{ "height": 420, "caption": "Both cards are always in the machine. Click any box, then flip the environment switch to see what changes and what does not.", "nodes": [ { "id": "phys0", "label": "Physical GPU 0", "kind": "host", "x": 0, "y": 40, "detail": "NVIDIA RTX 4000 Ada Generation at PCI 00000000:00:02.0, UUID GPU-14c73d7e-6cf9-0056-e332-5a4081954c17. The UUID and the slot address never change, whatever any mask says." }, { "id": "phys1", "label": "Physical GPU 1", "kind": "host", "x": 0, "y": 300, "detail": "NVIDIA RTX 4000 Ada Generation at PCI 00000000:00:03.0, UUID GPU-da3db8be-0267-da92-25a8-86d6655b77c5. This is the card the ticket asked for." }, { "id": "nvml", "label": "nvidia-smi (NVML)", "kind": "bmc", "x": 320, "y": 170, "detail": "The management path. NVML inventories every card the driver found and never reads CUDA_VISIBLE_DEVICES. Its two wires stay lit in both states, which is the point of the switch." }, { "id": "cuda0", "label": "CUDA device 0", "kind": "nic", "x": 660, "y": 20, "detail": "With no mask set, CUDA numbers the cards in driver order. Device 0 is physical GPU 0." }, { "id": "cuda1", "label": "CUDA device 1", "kind": "nic", "x": 660, "y": 170, "detail": "With no mask set, device 1 is physical GPU 1. Ordinals and physical positions agree, which is why the trap stays hidden until somebody masks something." }, { "id": "cudam", "label": "CUDA device 0 (masked)", "kind": "psu", "x": 660, "y": 320, "detail": "With CUDA_VISIBLE_DEVICES=1 the visible list holds one entry, physical GPU 1, and it is numbered from zero. The captured run says: device 0 [0000:00:03]." } ], "edges": [ { "from": "phys0", "to": "nvml", "label": "always listed", "kind": "mgmt" }, { "from": "phys1", "to": "nvml", "label": "always listed", "kind": "mgmt" }, { "from": "phys0", "to": "cuda0", "label": "device 0", "kind": "inband" }, { "from": "phys1", "to": "cuda1", "label": "device 1", "kind": "inband" }, { "from": "phys1", "to": "cudam", "label": "device 0", "kind": "inband" } ], "toggle": { "label": "Environment:", "on": "CUDA_VISIBLE_DEVICES not set", "off": "CUDA_VISIBLE_DEVICES=1", "dimOn": ["cudam", "e:phys1-cudam"], "dimOff": ["cuda0", "cuda1", "e:phys0-cuda0", "e:phys1-cuda1"] } }
Flip the switch and watch the two wires into nvidia-smi. They never dim. That is the finding you already produced yourself: masking is invisible from the management path, and only the CUDA path renumbers.
Stop for a second, because you just picked up a specific piece of authority.
When somebody tells you "the job is on GPU 0", you now have the follow up almost nobody asks: GPU 0 according to which tool. If they are quoting a CUDA log line written under a mask, that number is a position in a filtered list, not a card.
And you already know how to settle it for real. Your inventory command prints UUID and PCI address, the two identifiers no mask can renumber.
The value of the variable is a list, and the order inside it is not decoration. The first entry becomes device 0, the second becomes device 1, and so on down the list.
You have already seen the one entry case on real hardware. CUDA_VISIBLE_DEVICES=1 made physical card 1 into device 0.
So CUDA_VISIBLE_DEVICES=1,0 hands a CUDA process both cards in that stated order, with physical card 1 first. Written the other way round, 0,1, it hands over the same two cards in driver order.
Scaffolding comes off here. No command is shown for the rest of the lesson.
A colleague insists the reordered form is different, and that nvidia-smi will show the swap.
Settle it with evidence. Set the mask to 1,0 for one run and ask the driver for two fields only, index and uuid, in comma separated form.
prompt: ops@gpu-mask:~$ answer: CUDA_VISIBLE_DEVICES=1,0 nvidia-smi --query-gpu=index,uuid --format=csv output: index, uuid 0, GPU-14c73d7e-6cf9-0056-e332-5a4081954c17 1, GPU-da3db8be-0267-da92-25a8-86d6655b77c5 hint: Same shape as your masked query, with the value set to 1,0 and the field list cut down to index and uuid.
Index 0 is still the card whose UUID begins GPU-14c73d7e. The order you asked for had no effect on this output whatsoever.
That is the same rule showing up a third time, so here it is stated a third way. NVML answers the question "what hardware is in this machine". CUDA answers a different one: "what am I allowed to use, and what do I call it". Only the second question has anything to do with CUDA_VISIBLE_DEVICES.
Last position, and this one is the failure you will meet in a ticket.
>>> It fails, and the error never mentions the mask. An ordinal matching nothing produces an empty visible list, and a CUDA program with an empty list reports honestly that it can see no devices. The wording sends people to the driver, the kernel module and the card. Nothing there is wrong.
You will meet this error one day under pressure. Produce it on purpose now, while you already know the cause.
Run the same tiny benchmark with the same four flags, but mask to ordinal 7 on a machine that only has ordinals 0 and 1.
prompt: ops@gpu-mask:~$ answer: CUDA_VISIBLE_DEVICES=7 all_reduce_perf -b 8 -e 8 -f 1 -g 1 output:
#
localhost: Test CUDA failure util.cu:674 'no CUDA-capable device is detected' .. localhost pid 24017: Test failure common.cu:1454 hint: The benchmark command you already ran, with the mask value changed from 1 to 7.
Here is the line, and it is worth committing to memory:
localhost: Test CUDA failure util.cu:674 'no CUDA-capable device is detected'Both cards are healthy. The driver is loaded. nvidia-smi would list both of them, cheerfully, at the same moment this error prints.
The message is technically true and practically misleading. From inside that process there really is no CUDA capable device, because the mask filtered every one of them out. The process cannot see the cause of its own blindness.
So when this error lands in front of you, check in this order:
1. Read the value of CUDA_VISIBLE_DEVICES in the same shell or container that failed. An out of range ordinal, a stray space, or an empty value are all common. 2. Run your inventory command and confirm the driver still lists the hardware. 3. Only then start suspecting the driver, the kernel module or the card.
Step 1 costs three seconds and resolves this more often than the other two put together.
You walked in able to set a variable. You walk out able to explain why the tool you trusted disagreed with it, and which of the two to believe.
nvidia-smi --query-gpu=index,name,uuid,pci.bus_id --format=csv # inventory: what is physically here
CUDA_VISIBLE_DEVICES=1 nvidia-smi --query-gpu=index,name,uuid --format=csv # proof NVML ignores the mask
CUDA_VISIBLE_DEVICES=1 all_reduce_perf -b 8 -e 8 -f 1 -g 1 # what CUDA sees under the maskAnd the readings that matter:
Time to do it on real hardware. The machine in the bay below is the same two card box, with the driver loaded and the same benchmark on your PATH.
Five objectives close this ticket:
1. File the inventory. Index, name, UUID and PCI address for both cards, saved to ~/mask-baseline.txt. 2. Prove NVML ignores the mask. The masked query, saved to ~/mask-smi-ignored.txt, with both cards still in it. 3. Get the CUDA answer. A masked benchmark run saved to ~/mask-cuda-view.txt, showing which device number it used. 4. Reproduce the failure. The out of range run saved to ~/mask-error.txt. Capture error output as well as normal output, or the message will not land in the file. 5. The challenge. Deliver ~/mask-verdict.txt, a written verdict a colleague could act on. No hints on this one.
Here is the deal: the workspace shows no commands. It hands you one objective at a time and you recall the move. That recall is the point. It is how the last twenty minutes turn into a skill instead of a memory.
Stuck is normal. Hit Request a signal on any objective for a nudge, or ask Daemon. It knows which objective you are on and will nudge, not spoil.
Your machine gpu-mask is booting in the bay below: two NVIDIA RT
Practice The GPU That Moved in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.