Learn › GPU Operations & NCCL Troubleshooting › nvidia-smi Mastery
nvidia-smi -q -d PERFORMANCE - a hands-on Linux lab on a real virtual machine.
A slow GPU is almost never a mystery, and it is almost never heat. Read power, clocks and the Clocks Event Reasons field on real hardware, then cap a card to its 30 W floor and watch its clock fall about 91 percent with the reason code to prove it.
It is 09:12 on a Tuesday. The training job that finished in four hours yesterday is still running this morning. Same code, same data, same machine.
Someone in the channel has already decided why. The card is overheating. A second person agrees. A third suggests moving the box to a colder rack.
Nobody has looked. That is the part worth noticing.
A GPU is a graphics processing unit, the accelerator card that does the heavy arithmetic in machine learning. When one slows down, it almost always slows down for a reason the driver already knows.
Heat is one line on a list of nine. It is the line people reach for because it is the one they can picture. Two of those nine lines are about heat. The other seven are not.
By the end of this lesson you will make a card name its own cause. Then you will throttle one on purpose and watch its clock fall by about 91 percent in a single command.
This is a Practice Zone. The black boxes below are a safe sandbox that checks one command at a time, and the diagram reacts to you: click the boxes, flip the switch. The real machine comes at the end, under Ready to practice.
nvidia-smi is the NVIDIA System Management Interface, the command that ships with every NVIDIA driver. It is the window into every NVIDIA card on the machine.
Four numbers settle almost every slow-GPU argument. Learn what each one is before you read any of them.
Power draw, in watts. A watt is a unit of power, how fast the card is turning electricity into work. This card idles near 30 W.
Power limit, in watts. The ceiling the driver enforces. The card is not allowed past it.
SM clock, in megahertz. A clock is how many times per second the chip steps forward, and megahertz means millions of steps per second. SM stands for streaming multiprocessor, the part of the card that does the actual math. When a GPU slows down, this is the number that moved.
Temperature, in C. The one everybody guesses at, and the easiest of the four to rule out.
Read ceilings before readings, always. A power draw of 61 W means nothing until you know whether the limit is 70 W or 700 W.
There is a field that names the cause outright. Nearly everybody calls it the throttle reasons field. NVIDIA does not.
In the human readable report the section header reads Clocks Event Reasons. In the machine readable query the same data is called clocks_throttle_reasons.active. One tool, one driver, two names for one thing.
This is not trivia. Search the report for the word throttle and you find nothing. Conclude the field is missing, and you are back to guessing about heat. Look for Clocks Event Reasons and it is right there.
The same tool does this a second time, with clock names. You ask for clocks.sm and the driver answers with a column headed clocks.current.sm. Short name in, canonical name out. Watch for it in the next step.
Take a position first. Guessing and being wrong is what makes the real answer stick.
>>> There is no first suspect. The driver keeps a flag for every reason that is currently holding the clock down, and reports them together. Two can be active at the same time. That is exactly why this field is worth reading before you form an opinion, and why the person who reads it wins the argument.
Start with what this card is allowed to do.
--query-gpu asks the driver for named fields, in the order you list them. --format=csv prints them as comma separated values with a header row on top.
Seven fields answer the ceiling question completely. Type this exact command in the practice terminal and press Enter:
nvidia-smi --query-gpu=index,power.draw,power.limit,power.max_limit,clocks.sm,clocks.max.sm,temperature.gpu --format=csv
prompt: ops@gpu-power:~$ answer: nvidia-smi --query-gpu=index,power.draw,power.limit,power.max_limit,clocks.sm,clocks.max.sm,temperature.gpu --format=csv output: index, power.draw [W], power.limit [W], power.max_limit [W], clocks.current.sm [MHz], clocks.max.sm [MHz], temperature.gpu 0, 30.77 W, 130.00 W, 130.00 W, 2175 MHz, 3105 MHz, 41 1, 28.83 W, 130.00 W, 130.00 W, 2175 MHz, 3105 MHz, 40 hint: One tool, one long flag. --query-gpu= then the seven field names joined by commas with no spaces, then --format=csv.
Two cards, seven columns, and a header row that already taught you something.
You asked for clocks.sm. The driver answered with clocks.current.sm [MHz]. That is the naming quirk from two steps ago, live. The short form is what you type. The canonical form is what comes back.
Now the columns, left to right:
Two ceilings and two current values. That pairing is the whole discipline. 30.77 against 130.00, and 2175 against 3105, is a quiet card with room to move.
The limit has a top. It also has a bottom, and the bottom is what makes the rest of this lesson possible.
-q prints the full report for every card. -d POWER narrows it to the power section. Filtering that with grep gives the range in one line:
nvidia-smi -q -d POWER | grep -Ei "power limit|min power|max power" | head -12
On this machine that prints:
GPU Ceiling Power Limit
Current Power Limit : 130.00 W
Requested Power Limit : 130.00 W
Default Power Limit : 130.00 W
Min Power Limit : 30.00 W
Max Power Limit : 130.00 W
Current Power Limit : N/A
Requested Power Limit : N/A
Default Power Limit : N/A
Min Power Limit : N/A
Max Power Limit : N/A
GPU Ceiling Power LimitThe first five lines belong to GPU 0. Min Power Limit : 30.00 W and Max Power Limit : 130.00 W are the two ends of the adjustable range. Default Power Limit : 130.00 W is where the card sits until somebody moves it.
The rest is the report bleeding through, and it is worth understanding rather than ignoring. -q walks every card, so a grep across it interleaves both of them plus a second block of fields that do not apply to this hardware, printed as N/A. The trailing GPU Ceiling Power Limit is GPU 1 section beginning.
That is the trap with grepping -q output on a multi-GPU box. You get lines from every card with nothing saying which. Add -i 0 to aim a command at one card and the ambiguity disappears.
Here is the reading everybody argues about. Take a position before you see the rest.
>>> No. 61 W against a 130 W ceiling is under half the budget, and 45 C is a warm laptop. Nothing in that reading points at heat. This is the most common misread in GPU operations: the job is slow, the card feels warm, and heat gets the blame. On this hardware, in a short run, thermal throttling is simply not what happens.
Now the same cards with real work on them.
Fewer fields this time, and one new one. utilization.gpu is the percentage of the last sampling window in which the card had work running.
--format=csv,noheader drops the header row so the output is nothing but data. That is what you want when you are watching a number move.
Type this exact command:
nvidia-smi --query-gpu=index,power.draw,clocks.sm,temperature.gpu,utilization.gpu --format=csv,noheader
prompt: ops@gpu-power:~$ answer: nvidia-smi --query-gpu=index,power.draw,clocks.sm,temperature.gpu,utilization.gpu --format=csv,noheader output: 0, 61.48 W, 2355 MHz, 45, 100 % 1, 60.21 W, 2385 MHz, 45, 100 % hint: Same tool, same --query-gpu flag. Five fields: index, power.draw, clocks.sm, temperature.gpu, utilization.gpu. Then --format=csv,noheader.
Five numbers per card, and every one of them is evidence.
100 % on both cards. They are busy, so this is a real load and not an idle machine.
Now hold those readings against the ceilings you captured three steps ago:
These cards are not thermally throttling. You can say that with numbers behind it, which is more than the channel had.
But notice what you just did. You inferred a cause from four readings. That is a good habit and it is still an inference. The driver can do better. It can name the cause outright, and refuse to name the ones that are not happening.
-d PERFORMANCE narrows the full report to the performance section, which is where the reasons live:
nvidia-smi -q -d PERFORMANCE
Here is the GPU 0 section of that report, taken on this machine before the load run, with the card awake and nothing wrong:
GPU 00000000:00:02.0
Performance State : P0
Clocks Event Reasons
Idle : Not Active
Applications Clocks Setting : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
HW Thermal Slowdown : Not Active
HW Power Brake Slowdown : Not Active
Sync Boost : Not Active
SW Thermal Slowdown : Not Active
Display Clock Setting : Not Active
Clocks Event Reasons Counters
SW Power Capping : 0 us
Sync Boost : 0 us
SW Thermal Slowdown : 0 us
HW Thermal Slowdown : 0 us
HW Power Braking : 0 us
Sparse Operation Mode : N/ANine reasons, every one Not Active. Five counters, every one 0 us. That is the shape of a card with nothing holding it back, and it is worth capturing on a good day so you have something to compare against on a bad one.
The nine, in the order the driver prints them:
Two of the nine are about heat. Read that again the next time a channel decides a card is overheating.
The counters underneath are cumulative microseconds. SW Power Capping : 0 us means this card has spent zero time power capped since the driver loaded. A counter that is not zero is a card with a history, even if it looks fine right now.
Nine names is a lot to hold. The picture is simpler than the list.
Every reason is a separate hand on the same lever, and that lever is the SM clock. Any one of them can pull it down on its own. Flip the switch to compare the clean card you just read with the capped card coming up.
{ "height": 400, "caption": "Every reason the driver can report, and the one thing they all control. Click a box to read its line, then flip the run switch.", "nodes": [ { "id": "swcap", "label": "SW Power Cap", "kind": "psu", "x": 0, "y": 20, "detail": "A power limit is holding the clocks down. On the capped run this printed: SW Power Cap : Active, with the counter SW Power Capping : 33516786 us. This is the only reason that was active." }, { "id": "hwthermal", "label": "HW Thermal Slowdown", "kind": "sensor", "x": 0, "y": 140, "detail": "Hardware cut the clocks because of heat. On both captured runs this printed Not Active, at 45 to 46 C under sustained load. This is the one everybody guesses." }, { "id": "hwbrake", "label": "HW Power Brake Slowdown", "kind": "psu", "x": 0, "y": 260, "detail": "An external power brake signal pulled the clocks down. Printed Not Active on every capture from this machine." }, { "id": "swthermal", "label": "SW Thermal Slowdown", "kind": "sensor", "x": 290, "y": 20, "detail": "Software cut the clocks because of heat. The second of the two heat reasons, and also Not Active on every capture here." }, { "id": "appclocks", "label": "Applications Clocks Setting", "kind": "default", "x": 290, "y": 140, "detail": "Somebody pinned the clocks with a setting. Not a fault, a decision. Worth checking before you blame the hardware." }, { "id": "syncboost", "label": "Sync Boost", "kind": "default", "x": 290, "y": 260, "detail": "This card is clock matched to a group of cards and follows the slowest one. One sick card can pull a whole group down." }, { "id": "smclk", "label": "SM clock", "kind": "host", "x": 620, "y": 140, "detail": "The number that actually moves. 2355 MHz under load on this machine, 210 MHz once GPU 0 was capped at 30 W. Ceiling on this card is 3105 MHz." } ], "edges": [ { "from": "swcap", "to": "smclk", "label": "pulls down", "kind": "power" }, { "from": "hwthermal", "to": "smclk", "label": "pulls down", "kind": "plain" }, { "from": "hwbrake", "to": "smclk", "label": "pulls down", "kind": "power" }, { "from": "swthermal", "to": "smclk", "label": "pulls down", "kind": "plain" }, { "from": "appclocks", "to": "smclk", "label": "pulls down", "kind": "plain" }, { "from": "syncboost", "to": "smclk", "label": "pulls down", "kind": "plain" } ], "toggle": { "label": "Run:", "on": "GPU 0 capped at 30 W (one reason active)", "off": "Clean card (nothing active)", "dimOn": ["hwthermal", "hwbrake", "swthermal", "appclocks", "syncboost", "e:hwthermal-smclk", "e:hwbrake-smclk", "e:swthermal-smclk", "e:appclocks-smclk", "e:syncboost-smclk"], "dimOff": ["swcap", "hwthermal", "hwbrake", "swthermal", "appclocks", "syncboost", "e:swcap-smclk", "e:hwthermal-smclk", "e:hwbrake-smclk", "e:appclocks-smclk", "e:swthermal-smclk", "e:syncboost-smclk"] } }
Notice what the switch does not change. The suspects are always all there. Only one of them is ever lit, and reading which one is the entire job.
Last position to take, and this one is about mechanism.
>>> It collapses. This is the mechanism to carry out of this lesson. A power limit is not a wall the card bumps into. It is a target the driver hits by lowering the clock until the watts fit underneath. Watts are the request. The clock is the lever. Nothing errors and nothing warns, which is exactly why a capped card is so easy to misdiagnose.
Reading a healthy card teaches you the shape of healthy. To learn the shape of a throttle you have to cause one.
A power cap is the safest throttle there is. Reversible in one command, no heat, no risk to the hardware.
-pl sets the power limit, in watts. -i 0 aims the command at GPU 0 and leaves GPU 1 alone. Changing a limit changes hardware behavior, so it needs root, which is what sudo is for.
Type this exact command:
sudo nvidia-smi -i 0 -pl 30
prompt: ops@gpu-power:~$ answer: sudo nvidia-smi -i 0 -pl 30 ||| nvidia-smi -i 0 -pl 30 output: Power limit for GPU 00000000:00:02.0 was set to 30.00 W from 130.00 W. All done. hint: sudo, then the tool, then -i 0 to aim at GPU 0, then -pl with the wattage you want.
The driver reports both ends of the change, to 30.00 W from 130.00 W, and All done. confirms the write landed. It also named the card by PCI address, 00000000:00:02.0, not by index. Indexes can move between boots. PCI addresses do not.
Four fields on one line: what the card is drawing, what it is allowed to draw, where its clock is, and the reason code.
clocks_throttle_reasons.active is the machine readable form of the Clocks Event Reasons section. It prints one hexadecimal number instead of nine lines of English.
Type this exact command:
nvidia-smi -i 0 --query-gpu=power.draw,power.limit,clocks.sm,clocks_throttle_reasons.active --format=csv,noheader
prompt: ops@gpu-power:~$ answer: nvidia-smi -i 0 --query-gpu=power.draw,power.limit,clocks.sm,clocks_throttle_reasons.active --format=csv,noheader output: 11.31 W, 30.00 W, 210 MHz, 0x0000000000000004 hint: Aim at GPU 0 with -i 0, then --query-gpu= with four fields: power.draw, power.limit, clocks.sm, clocks_throttle_reasons.active. Then --format=csv,noheader.
There it is.
From 2355 MHz to 210 MHz. That is 2145 MHz gone, about 91 percent of the clock, from one command, with work still queued on the card.
Two things in that table deserve a second look.
210 MHz is not a random number. It is the floor this card drops to when it has nothing to do at all. Hold onto it, because it comes back later in this lesson from a completely different run.
11.31 W against a 30.00 W limit. The card is not even touching its new ceiling. The driver did not wait for the cap to be hit and then react. It cut the clock in advance so the cap could not be exceeded. Preventive, not reactive, which is why a capped card looks so calm while it underperforms.
And the code. 0x0000000000000004 is one hexad
Practice The Card That Slowed Down in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.