Learn › GPU Operations & NCCL Troubleshooting › nvidia-smi Mastery
nvidia-smi - a hands-on Linux lab on a real virtual machine.
Your first GPU command. Run nvidia-smi and read the whole cockpit: driver and CUDA versions, temperature, power, memory, utilization, and the process list, on a real RTX 4000 Ada capture. Then pull single facts with --query-gpu. Free preview lesson.
Your team just got its first GPU server. Someone paid real money for the card inside it, the driver install said it finished, and now everyone is looking at you. Is the GPU actually there? Is it alive? Is anything using it?
On every NVIDIA machine on Earth, from a hobby box to a 10,000-GPU training cluster, that question has the same one-word answer. You type nvidia-smi and the machine shows you its cockpit. This lesson teaches you to read every gauge on it.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. The outputs are one real capture from an NVIDIA RTX 4000 Ada, the exact card our hands-on GPU labs run on. On live hardware your temperatures, clocks, and memory numbers will differ; the shape of the output will not.
nvidia-smi is short for NVIDIA System Management Interface. It is a command-line program that ships with every NVIDIA driver. It reports what the driver knows: which GPUs exist, how hot they run, how much memory they use, and which programs are on them.
One idea makes everything about this tool click: nvidia-smi never talks to the GPU directly. It talks to the driver, the piece of software the operating system loads to control the card. Click through the chain below; each hop is a place where things can break.
{ "height": 260, "caption": "nvidia-smi asks the driver. The driver asks the card. Click each hop.", "nodes": [ { "id": "you", "label": "Your terminal", "kind": "admin", "x": 0, "y": 130, "detail": "You type nvidia-smi at a shell prompt, like any other Linux command. No flags needed for the default dashboard." }, { "id": "smi", "label": "nvidia-smi", "kind": "host", "x": 300, "y": 130, "detail": "The tool itself. It ships with the NVIDIA driver package and reads GPU state through a library called NVML. If the shell says command not found, the driver package was never installed on this machine." }, { "id": "kmd", "label": "NVIDIA driver", "kind": "nic", "x": 600, "y": 130, "detail": "The kernel driver (the KMD, kernel-mode driver). The operating system loads it at boot. If it is missing or failed to load, nvidia-smi prints its famous could-not-communicate error even when the card is perfectly healthy." }, { "id": "gpu", "label": "GPU (RTX 4000 Ada)", "kind": "psu", "x": 900, "y": 130, "detail": "The physical card on the PCIe bus. Even with no driver at all, the hardware still shows up in lspci, which is how you tell a driver problem from a missing card." } ], "edges": [ { "from": "you", "to": "smi", "label": "you run it", "kind": "inband" }, { "from": "smi", "to": "kmd", "label": "asks via NVML", "kind": "mgmt" }, { "from": "kmd", "to": "gpu", "label": "PCIe", "kind": "power" } ] }
Hold onto the chain: you, then the tool, then the driver, then the card. When nvidia-smi works, it proves the whole chain works. That is why it is the first command you run on any GPU machine: one command, and the entire stack is confirmed alive.
Time for first contact. Type the command exactly as shown and press Enter:
nvidia-smi
prompt: student@gpu-lab:~$ answer: nvidia-smi output: Mon Jun 15 03:18:29 2026 +-----------------------------------------------------------------------------------------+
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
+-----------------------------------------------------------------------------------------+ hint: Type the twelve characters nvidia-smi and press Enter. Dash included, no flags.
That wall of boxes is the whole story of this machine's GPU, told in three sections: a header, a GPU table, and a process list. It looks dense now. By the end of this lesson you will read it in five seconds, the way you already glance at a car dashboard.
The top line inside the frame identifies the software side of the chain:
| NVIDIA-SMI 610.43.02 KMD Version: 610.43.02 CUDA UMD Version: 13.3 |
Three facts live here. NVIDIA-SMI 610.43.02 is the version of the tool itself. KMD Version is the kernel driver version, the hop from the diagram. CUDA UMD Version is the highest CUDA version this driver can support, which is the first thing you check when someone asks if the box can run their framework.
Older guides and older machines print Driver Version and CUDA Version here instead. NVIDIA renamed both fields in the 610 driver series (KMD means kernel-mode driver, UMD means user-mode driver). Same facts, newer labels. If you see the old names on a machine at work, you are simply on an older driver.
The middle table is the heart of the dashboard. Its column headers stack three lines deep, and each header line describes the matching line of the GPU row. This machine has one GPU, so there is one row block:
| 0 NVIDIA RTX 4000 Ada Gene... Off | 00000000:00:02.0 Off | Off |
| 30% 42C P0 33W / 130W | 0MiB / 20475MiB | 0% Default |
First line, left to right: GPU index 0 (the number you use to target this card), the model name (truncated with ... to fit), and Persistence-M Off. Then the Bus-Id, the card's PCIe address, and Disp.A Off, which says no display is attached; this is a compute card, not a gaming rig. ECC Off closes the line.
Second line is the live health readout, and it is the line you will glance at for the rest of your career. Fan at 30%. Temperature 42C, a comfortable idle. Performance state P0. Power draw 33W of a 130W ceiling. Memory: 0MiB used of 20475MiB total, so 20 GB free. GPU utilization 0%.
Put the numbers together and this row tells one coherent story: a healthy, cool, completely bored GPU with nothing to do. Learn that story shape. A busy-training-run row tells a different story with the same fields: high utilization, memory nearly full, power near the cap, temperature climbing.
About P0: performance states run from P0 (maximum clocks) down through P8 and lower (deep power saving). Datacenter-class cards like this one often sit at P0 even when idle; the fine print of why belongs to the deep-query lesson later in this module. What matters today: Perf plus Pwr plus GPU-Util together tell you how hard the card is working.
The bottom section answers the question you will be asked most often: *who is using the GPU right now?*
| No running processes found |
Nothing is on the card. When jobs are running, this table shows one line per process: which GPU it is on, its PID (the process ID you already know from Linux), and how much GPU memory it holds. When a teammate says the GPU is full but my job is not running, this table is where the mystery ends; some process is holding memory, and the PID tells you whose.
The dashboard is for humans. Scripts and monitoring systems want single facts in a parseable shape, and nvidia-smi has a second mode for exactly that: --query-gpu picks the fields, --format=csv prints them as comma-separated values.
Ask this card for its name, total memory, and temperature:
nvidia-smi --query-gpu=name,memory.total,temperature.gpu --format=csv
prompt: student@gpu-lab:~$ answer: nvidia-smi --query-gpu=name,memory.total,temperature.gpu --format=csv output: name, memory.total [MiB], temperature.gpu NVIDIA RTX 4000 Ada Generation, 20475 MiB, 42 hint: The flags are --query-gpu=name,memory.total,temperature.gpu and --format=csv, separated by a space. No spaces inside the field list.
Same machine, same facts, radically friendlier to software: a header row naming each field, then one row per GPU. Notice the full model name appears here with no ... truncation; the dashboard cut it off for space, the query mode never does. Nearly every GPU monitoring dashboard you will ever see is built on queries like this one.
A script needs this machine's driver version, nothing else: no header line, no extra fields, just the value. The field is called driver_version, and --format=csv,noheader drops the header row. Build the command yourself.
prompt: student@gpu-lab:~$ answer: nvidia-smi --query-gpu=driver_version --format=csv,noheader output: 610.43.02 hint: Start from the query form you just ran. One field: driver_version. Then add ,noheader after csv, with no space around the comma.
One command, one naked value, ready for a script to consume. You have now used both faces of nvidia-smi: the human dashboard and the machine query. Every flag you meet later in this module builds on these two.
On a healthy machine you get the dashboard. On an unhealthy one, nvidia-smi fails in two famous ways, and the chain diagram tells you exactly which hop broke.
Error one: NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running. The tool exists but the driver hop is down. The driver never installed properly, failed to load at boot, or a kernel update broke it. Check first: lsmod | grep nvidia to see whether the driver module is loaded at all.
Error two: nvidia-smi: command not found. The shell cannot even find the tool, which means the NVIDIA driver package was never installed on this machine; the tool ships with it. Check first: does the hardware exist? lspci | grep -i nvidia talks to the PCIe bus directly and lists the card even with zero NVIDIA software installed.
Notice how the chain sorts the panic for you. Command not found means no software. Could-not-communicate means software present, driver hop broken. Dashboard prints but a card is missing from the table: now it is a hardware or bus question, and lspci is your witness.
Twenty minutes ago this was a wall of boxes. Now every region has a meaning you can say out loud:
A few doors stay deliberately closed today; each gets its own lesson later in this module. nvidia-smi -L lists every GPU with its unique UUID. -q dumps the full deep-query report the dashboard summarizes. dmon streams live per-second readings. And engineers everywhere wrap the habit in watch -n 1 nvidia-smi to turn the snapshot into a live view.
This preview is the first lesson of the nvidia-smi Mastery module, rebuilt to the same standard as our Linux Foundations track. The hands-on version boots you a real cloud server with a real RTX 4000 Ada, where you run these commands against live silicon and watch your own numbers move.
GPU labs go live on August 12. The rest of this module, deep queries, live monitoring, topology, and GPU management, arrives with it. If you want the Linux side of the job in the meantime, the Linux Foundations track is open now and starts from zero.
Practice First Contact: GPU Detected, Driver Loaded in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.