Five of the twenty two fields in the NVMe SMART log decide whether you replace a drive. Here is what every line means in the order nvme-cli prints it, which numbers are odometers, and the exact values that mean act now.
Five fields decide whether you act. critical_warning, where any nonzero bit is a live alarm. percentage_used, the wear estimate allowed to pass 100. available_spare against available_spare_threshold, the real end of life line. media_errors, data actually harmed. And the thermal time counters, how long the drive ran hot. The rest of the log is context.
The healthy output below was captured on a production Amazon Linux 2023 host with Nitro NVMe. Every degraded value here is quoted from vendor documentation or a public operator report, with a source line attached, because our drives are healthy and we will not fake a dying one.
Install nvme-cli and smartmontools from your package manager (dnf install nvme-cli smartmontools, or apt install on Debian), then check the version: labels and decoding changed between the 1.x and 2.x series.
root@storage-01:~# nvme version
nvme version 2.13 (git 2.13)
libnvme version 1.13 (git 1.13)nvme list tells you which node is which drive before you go reading logs from the wrong one.
root@storage-01:~# nvme list
Node Generic SN Model Namespace Usage Format FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
/dev/nvme0n1 /dev/ng0n1 vol01bfe3213eee274a1 Amazon Elastic Block Store 0x1 53.69 GB / 53.69 GB 512 B + 0 B 1.0 The serial is the EBS volume id with the dash removed, which is how AWS documents the mapping from a device node back to a volume. On real hardware that column carries the manufacturer serial instead.
Two device forms matter. /dev/nvme0 is the controller character device, /dev/nvme0n1 a namespace block device. The SMART log is a controller level page, requested for namespace ffffffff, and per namespace support is optional. On this controller the namespace form is refused.
root@storage-01:~# nvme smart-log /dev/nvme0n1
smart log: No such device or addressThat is not a broken install. It is an optional feature the controller does not implement. Read the log from the character device and the failure disappears.
This is the whole log page, log identifier 02h, in the order nvme-cli 2.13 prints it.
root@storage-01:~# nvme smart-log /dev/nvme0
Smart Log for NVME device:nvme0 namespace-id:ffffffff
critical_warning : 0
temperature : -273 °C (0 K)
available_spare : 0%
available_spare_threshold : 0%
percentage_used : 0%
endurance group critical warning summary: 0
Data Units Read : 0 (0.00 B)
Data Units Written : 0 (0.00 B)
host_read_commands : 0
host_write_commands : 0
controller_busy_time : 0
power_cycles : 0
power_on_hours : 0
unsafe_shutdowns : 0
media_errors : 0
num_err_log_entries : 0
Warning Temperature Time : 0
Critical Composite Temperature Time : 0
Thermal Management T1 Trans Count : 0
Thermal Management T2 Trans Count : 0
Thermal Management T1 Total Time : 0
Thermal Management T2 Total Time : 0Everything is zero, and that is the honest result on Elastic Block Store: the virtual controller answers the Get Log Page command and populates none of the counters. AWS documents the identity fields and a separate performance stats plugin for these devices, and does not document log page 02h at all. Take this as proof of shape, not of wear. Note available_spare 0% beside available_spare_threshold 0%, which on real hardware would read as a finished drive. It raises no warning bit, because bit 0 fires when spare falls below the threshold, and zero is not below zero.
Field by field, in print order:
critical_warning is one byte of independent flags. Zero is the only valuethat means nothing is wrong. Decoded next.
temperature is Composite Temperature for the whole device, Kelvin on thewire. It is a vendor defined blend, not a probe at one point.
available_spare is the reserve blocks remaining, normalized 0 to 100.available_spare_threshold is the vendor floor for that field. Real valuesare usually 5 or 10 percent.
percentage_used is the vendor wear estimate. Covered below, including whyit can read 133.
endurance group critical warning summary is the same bits rolled up perendurance group. Zero on almost every client drive.
Data Units Read and Data Units Written are lifetime host transfer, in astrange unit that gets its own section.
host_read_commands and host_write_commands count commands, not bytes.Data units divided by these gives your average I/O size.
controller_busy_time is minutes with at least one command outstanding.power_cycles counts power cycles.power_on_hours excludes low power states, so on a laptop it lags wall clock.unsafe_shutdowns increments when power went away before the shutdownnotification. Climbing on a server means a power problem, and each event is a chance for the translation layer to lose a write.
media_errors counts unrecovered data integrity errors: uncorrectable ECC,CRC failure, LBA tag mismatch. Data was hurt.
num_err_log_entries counts lifetime entries in the error log, page 01h,which includes ordinary rejected commands. It is not a body count.
Warning Temperature Time and Critical Composite Temperature Time arelifetime minutes in each temperature band.
Thermal Management T1 Trans Count and T2 Trans Count count entries intolight and heavy throttling.
Thermal Management T1 Total Time and T2 Total Time measure how long itstayed there, in seconds, not minutes.
Four unit systems in one screen: Kelvin, minutes, seconds, and blocks of a thousand sectors. Misreading one of them is the most common way to get this log wrong.
For scripts, ask for JSON. The keys are stable, and they are not the display labels, which is why searching for warning_temp_time finds nothing in the plain text output.
root@storage-01:~# nvme smart-log /dev/nvme0 -o json
{
"critical_warning":0,
"temperature":0,
"avail_spare":0,
"spare_thresh":0,
"percent_used":0,
"endurance_grp_critical_warning_summary":0,
"data_units_read":0,
"data_units_written":0,
"host_read_commands":0,
"host_write_commands":0,
"controller_busy_time":0,
"power_cycles":0,
"power_on_hours":0,
"unsafe_shutdowns":0,
"media_errors":0,
"num_err_log_entries":0,
"warning_temp_time":0,
"critical_comp_time":0,
"thm_temp1_trans_count":0,
"thm_temp2_trans_count":0,
"thm_temp1_total_time":0,
"thm_temp2_total_time":0
}Any nonzero value means the controller is asserting a failure state right now. The bits are independent, several can be set at once, and the value you read is their sum. Modern nvme-cli will name them for you with -H, which is not in the man page for this subcommand but works in 2.13. The rest of the log follows below the decode, unchanged:
root@storage-01:~# nvme smart-log /dev/nvme0 -H
Smart Log for NVME device:nvme0 namespace-id:ffffffff
critical_warning : 0
Available Spare[0] : 0
Temp. Threshold[1] : 0
NVM subsystem Reliability[2] : 0
Read-only[3] : 0
Volatile mem. backup failed[4] : 0
Persistent Mem. RO[5] : 0Those six names come straight from the specification. The bits, their masks and what they mean in practice:
| Bit | Mask | Meaning | What it means in practice |
|---|---|---|---|
| 0 | 0x1 | Available spare below threshold | Out of reserve blocks. The real end of life signal. Replace it. |
| 1 | 0x2 | Temperature past a warning or critical threshold | A live thermal alarm. It clears when the drive cools, because these bits are state, not history. |
| 2 | 0x4 | NVM subsystem reliability degraded | Two very different causes. See below. |
| 3 | 0x8 | Media placed in read only mode | Writes are gone. Evacuate the data now. |
| 4 | 0x10 | Volatile memory backup device failed | Only on drives with power loss protection. Replacement territory. |
| 5 | 0x20 | Persistent Memory Region read only or unreliable | Rare, only on controllers with a PMR. |
Bit assignments per the NVM Express Base Specification and the libnvme nvme_smart_crit enum, https://manpages.ubuntu.com/manpages/jammy/man2/nvme_smart_crit.2.html. Bit 5 was added in NVMe 1.4. Bits 6 and 7 are reserved.
Read the sums the way you read file permissions. 0x1 is spare exhaustion alone. 0x4 is degraded reliability alone, and it is the value in nearly every report you will find online. 0x5 is both at once, which is a drive that is genuinely finished.
percentage_used is an estimate, not a measurement. The specification says 100 means the rated endurance has been consumed, that this does not itself indicate failure, that the value is allowed to exceed 100, that anything above 254 displays as 255, and that it updates once per power on hour. Solidigm is blunter: a value between 1 and 99 is acceptable, 100 is not. Kingston calls a drive past 100 suspect and leaves the decision to the operator.
So percentage_used: 133 is not a bug. It is a drive running 33 percent past the endurance the manufacturer rated and warranted. What tells you whether it is failing is the spare pair above it.
available_spare is the reserve pool the controller retires blocks into. While it sits above available_spare_threshold, the drive still has somewhere to put failures. When it drops below, bit 0 sets, and that is the alarm worth waking up for. Watch the trend, not the number: a spare that fell from 100 to 95 in a week is louder than one parked at 80 for two years.
The data units are what people get wrong most often. One unit is a thousand 512 byte units, so 512,000 bytes, and modern nvme-cli prints the conversion. From the drive quoted below: 42,210,392 x 512,000 = 21,611,720,704,000 bytes, the 21.61 TB the tool prints, in decimal terabytes. That is how you compare Data Units Written against the terabytes written rating on the datasheet.
Here is what those fields look like on a drive that has done work, since ours reports zeros:
Source: linux-nvme/nvme-cli issue 2456, nvme-cli 2.10.2 with libnvme 1.10 (abridged)
https://github.com/linux-nvme/nvme-cli/issues/2456
available_spare : 100%
available_spare_threshold : 5%
percentage_used : 1%
Data Units Read : 42,210,392 (21.61 TB)
Data Units Written : 20,246,705 (10.37 TB)
power_on_hours : 5,221
unsafe_shutdowns : 78
media_errors : 0
num_err_log_entries : 1,124Note the last two lines together. Over a thousand error log entries, zero media errors, and the drive is healthy. Those two counters answer different questions.
Composite temperature is Kelvin on the wire, and the two temperature time counters only work if the controller published thresholds. Read those from the controller identity first with nvme id-ctrl /dev/nvme0 | grep -iE "wctemp|cctemp". On the controller captured here it answers wctemp : 343 and cctemp : 0.
wctemp 343 is the warning threshold, 343 K, which is 70 C. cctemp 0 means the critical threshold is unset, and the specification is explicit that when a threshold is 0h the matching time counter reads 0 no matter how hot the drive gets. So here Critical Composite Temperature Time is dead by construction, not by good luck. smartctl converts the same field, one line out of a much longer report:
root@storage-01:~# smartctl -d nvme -a /dev/nvme0
Warning Comp. Temp. Threshold: 70 CelsiusThe current temperature reads back as 0 K, which nvme-cli renders as minus 273 Celsius. That is the unpopulated field again, not a cryogenic drive.
To confirm throttling on real hardware, treat the counters as odometers and diff them:
1. Read wctemp and cctemp once. If both are 0h, the time counters will never move and you are left with the temperature value itself. 2. Record Warning Temperature Time, Critical Composite Temperature Time and the four Thermal Management counters before the workload. 3. Run the workload that feels slow, read the log again, subtract. Growth in Warning Temperature Time means minutes spent at or above the warning threshold. Growth in T1 or T2 means the controller throttled itself, lightly then hard. Growth in Critical Composite Temperature Time is a cooling emergency. 4. Watch bit 1 of critical_warning during the run: it appears while the drive is hot and clears when it cools, because those bits are current state. If the thermal counters stay flat, heat is not your story, so check percentage_used and media_errors before blaming airflow.
Mind the unit split: the two temperature time counters are minutes, the two thermal management total times are seconds.
The same 0x4 has two opposite causes, and one output tells them apart. First, the benign kind, where the wear estimate simply passed 100:
Source: netdata discussion 17311, operator report on Hetzner drives
https://github.com/netdata/netdata/discussions/17311
Values reported in the thread:
critical_warning 0x04
percentage_used 133%
available_spare 100%
available_spare_threshold 10%
media_errors 0The operator guidance in that thread is that the warning means the warranty is over, and the drive is fine as long as available spare stays above its threshold. A hosting provider reached the same conclusion for a Samsung PM981a rated at 200 TB written that had taken roughly 1000 TB (https://forum.proxmox.com/threads/how-to-get-rid-of-smart-reliability-notifications.130103/).
Now the same hex value, on a nearly new drive, meaning something completely different:
Source: Framework Community thread, WD_BLACK SN770 1TB
https://community.frame.work/t/smart-disk-monitor-device-dev-nvme0-critical-warning-0x04-reliability/83516
Values reported in the thread:
critical_warning 0x04
percentage_used 1%
available_spare 100%
media_errors 11
error log repeated Unrecovered Read Error entries at specific LBAsWear estimate at 1 percent, spare untouched, and eleven unrecovered read errors. That is media failure, not old age, and the thread ended in a warranty replacement. The discriminator between the two cases is always media_errors and percentage_used, never the warning byte alone.
And a third case, where the scary looking counter meant nothing at all:
Source: TrueNAS forum, smartctl on nvme and what the status code means
https://www.truenas.com/community/threads/smartctl-on-nvme-what-does-the-status-code-mean.86920/
Values reported in the thread:
critical_warning 0x04
available_spare 76% (threshold 10%)
percentage_used 3%
media_errors 0
num_err_log_entries 66
after a firmware update error count 0, smartctl -H passing againIf you want to know what those entries were, read them. The error log has fixed slots, so empty ones read back as zeros:
root@storage-01:~# nvme error-log /dev/nvme0 -e 4
Error Log Entries for device:nvme0 entries:4
.................
Entry[ 0]
.................
error_count : 0
sqid : 0
cmdid : 0
status_field : 0 (Successful Completion: The command completed without error)Fleet context, for calibration: Backblaze reports a lifetime annualized failure rate of 0.90 percent across its SSD boot drives, with an average age at failure of 14 months, and notes that SATA drives never agreed on attribute names the way this NVMe log page does (https://www.backblaze.com/blog/ssd-edition-2023-mid-year-drive-stats-review/).
smartctl -d nvme -a fetches the same log 02h and relabels it in English. Media and Data Integrity Errors is media_errors, and Error Information Log Entries is num_err_log_entries. Here it is with the identity section trimmed off the top:
root@storage-01:~# smartctl -d nvme -a /dev/nvme0
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: -
Available Spare: 0%
Available Spare Threshold: 0%
Percentage Used: 0%
Data Units Read: 0
Data Units Written: 0
Host Read Commands: 0
Host Write Commands: 0
Controller Busy Time: 0
Power Cycles: 0
Power On Hours: 0
Unsafe Shutdowns: 0
Media and Data Integrity Errors: 0
Error Information Log Entries: 0
Warning Comp. Temperature Time: 0
Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors LoggedTwo things to notice. Temperature prints as a bare dash rather than a nonsense figure, which smartctl 7.2 handles better than nvme-cli does. And this build prints fewer fields than nvme-cli: no endurance group summary, no thermal management counters.
The health verdict is the part to be careful with:
root@storage-01:~# smartctl -d nvme -H /dev/nvme0
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-6.1.163-186.299.amzn2023.x86_64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSEDFor NVMe, that PASSED or FAILED is read straight out of the critical warning byte and nothing else. That is why a drive with percentage_used: 133 and a perfect spare reserve reports FAILED in smartctl while serving traffic normally, and why the TrueNAS drive above flipped back to passing after a firmware update touched only its error log.
| Field | Meaning | Act when |
|---|---|---|
critical_warning | Bitfield of live failure states | Any nonzero value. Decode the bit first |
temperature | Composite temperature, Kelvin on the wire | At or above wctemp under normal load |
available_spare | Reserve blocks remaining, percent | Falling trend, or below the threshold |
available_spare_threshold | Vendor floor for the field above | Reference line, not an alarm |
percentage_used | Vendor wear estimate, may exceed 100 | 100 or more: plan replacement, watch spare |
endurance group critical warning summary | Same bits, per endurance group | Any nonzero value |
Data Units Read / Written | Lifetime host transfer, 512,000 bytes per unit | Compare written against the rated TBW |
host_read_commands / host_write_commands | Completed command counts | Context for average I/O size |
controller_busy_time | Minutes with I/O outstanding | Context for throughput math |
power_cycles | Power cycle count | Context |
power_on_hours | Powered hours, low power excluded | Context for every rate you calculate |
unsafe_shutdowns | Power lost before shutdown notification | Climbing on a server: fix the power path |
media_errors | Unrecovered data integrity errors | Any nonzero: check backups. Climbing: replace |
num_err_log_entries | Lifetime error log entries, page 01h | Only with media errors or a warning bit |
Warning Temperature Time | Minutes at or above wctemp |