Read the RAW_VALUE column for attributes 5, 197 and 198 and ignore the normalized 100s. Zero is the only comfortable number, growth between two readings is a replacement, and SAS disks answer the same question with a grown defect list instead.
Read RAW_VALUE for attributes 5, 197 and 198. Ignore the normalized VALUE, WORST and THRESH columns, which sit at 100 until it is far too late. Zero is the only comfortable number. Any of the three growing between two readings a week apart is a replacement, not a watch item.
Every command and every line of output below was captured from the Camp's own bare-metal server, on the enterprise Seagate disks in its backplane plus one Kingston SSD for contrast. The failure statistics are cited from Backblaze and from the Google disk population study, and are named where they appear.
Three separate verdicts, stacked. -H is the firmware's own opinion. -A is the attribute table, which is where the counts live. -l reads the logs. They disagree more often than you would like, and when they do, the attribute table and the logs win.
professorlinux@L:~# sudo smartctl -i /dev/sdl
Model Family: Seagate Constellation.2 (SATA)
Device Model: ST9250610NS
Serial Number: 9XE08VD1
User Capacity: 250,059,350,016 bytes [250 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
SMART support is: Enabled
professorlinux@L:~# sudo smartctl -H /dev/sdl
SMART overall-health self-assessment test result: PASSEDPASSED means the firmware has not tripped its own thresholds. It is not a health report. Here is the same disk's attribute table, trimmed to the rows that teach something.
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 075 063 --- Pre-fail Always - 36804852
5 Reallocated_Sector_Ct 0x0033 100 100 --- Pre-fail Always - 0
9 Power_On_Hours 0x0032 074 011 --- Old_age Always - 22980
190 Airflow_Temperature_Cel 0x0022 076 058 --- Old_age Always - 24 (Min/Max 22/28)
194 Temperature_Celsius 0x0022 024 042 --- Old_age Always - 24 (0 16 0 0 0)
195 Hardware_ECC_Recovered 0x001a 111 099 --- Old_age Always - 36804852
197 Current_Pending_Sector 0x0012 100 100 --- Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 --- Old_age Offline - 0VALUE, WORST and THRESH are a vendor-normalized score, not a count. smartctl(8) is explicit that smartctl reports the numbers as read from the device and does not convert between raw and normalized. An attribute has failed only when VALUE drops to or below THRESH, which is what fills the WHEN_FAILED column.
Four things in that block are worth staring at.
Attribute 1 reads 36804852, which looks like thirty-six million read errors. Attribute 195, Hardware_ECC_Recovered, reads exactly the same number. It is one counter surfaced twice: sectors read and corrected, not sectors lost. The smartmontools FAQ says the raw value of attribute 1 is essentially useless on Seagate drives for this reason. Do not build alerting on it.
THRESH is --- on every row. This Dell-branded Seagate publishes no thresholds at all, so the pre-fail mechanism on this disk is decorative. The raw counts are all you have.
Attributes 190 and 194 are both temperature and both raw 24, normalized two different ways on the same drive. 190 shows 076, roughly 100 minus the temperature. 194 shows 024, which is the temperature itself, so its "WORST" of 042 is the hottest it has ever been rather than the worst score.
Attribute 5 shows VALUE 100 and RAW 0. Those mean different things. A drive with VALUE 100 and RAW 12 has twelve remapped sectors and a perfect-looking score.
Reallocated_Sector_Ct counts sectors the drive found bad and permanently redirected to spares it has kept hidden since the factory. Capacity does not shrink, the filesystem never notices, and this counter is the only witness. Small and unchanged is a watch item. Growing is a replacement.
Attribute names are a smartctl database lookup, not a standard. The SSD in the same chassis renames the same ID:
professorlinux@L:~# sudo smartctl -A /dev/sdp
5 Retired_Block_Count 0x0033 100 100 003 Pre-fail Always - 0
196 Reallocated_Event_Count 0x0033 100 100 003 Pre-fail Always - 0
198 Uncorrectable_Sector_Ct 0x0010 120 120 000 Old_age Offline - 0/0That is a KINGSTON SE100S3200G. It publishes real thresholds where the Seagate published none, and it carries attribute 196, which the Seagate omits. 196 counts remap operations, 5 counts sectors, and one operation can move several sectors, so they drift apart without either being wrong. Judge by 5.
Current_Pending_Sector is not history. It is the count of sectors the drive tried to read, failed to read, and has not been able to resolve. Whatever was stored there is unreachable right now. This is the scariest of the three.
The drive will not remap a pending sector on its own, because remapping throws away the last chance of recovering what is in it. The smartmontools FAQ puts the exits precisely: the replacement does not happen until the disk can read the data on the bad sector, or is told to write new data to it. A later successful read clears the count with no reallocation. A write clears it too, either succeeding in place or forcing a remap that bumps attribute 5. That is why 197 can fall back to zero while 5 never moves, and why a 197 that keeps coming back is a disk with an opinion about its own future.
If your pending count is exactly 8, that is arithmetic, not coincidence. A 512e drive has 4096-byte physical sectors presented as eight 512-byte logical sectors. One dead physical sector takes all eight with it, so these drives count pending sectors in multiples of 8. The Seagate above reports Sector Size: 512 bytes logical/physical, a native 512-byte drive, so one bad sector there counts as 1.
No ATA disk on this host is failing, so what nonzero looks like is quoted rather than ours:
Source: smartmontools Bad Block HOWTO (https://www.smartmontools.org/wiki/BadBlockHowto)
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 1
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 1Reallocated is still 0 there. Every normalized VALUE still reads 100. Only the raw columns on 197 and 198 say anything happened.
Offline_Uncorrectable is what the drive's own background surface scan found and could not correct. Look at the UPDATED column on 198 in our capture: it says Offline, not Always, because that row only moves when a scan runs. Attribute 187, Reported_Uncorrect, is the other reporter: errors ECC could not fix that were handed back to the host. Both read zero on our Seagate, alongside 188:
187 Reported_Uncorrect 0x0032 100 100 --- Old_age Always - 0
188 Command_Timeout 0x0032 100 100 --- Old_age Always - 0187 is the one Backblaze treats as decisive. In their 2014 drive-stats post, Brian Beach reported that drives with zero uncorrectable errors hardly ever fail, and that once 187 goes above zero they schedule a replacement. 188 earns a second look before you condemn a disk, since a timeout can come from a cable, an expander or a controller rather than the platters.
SAS disks do not have a SMART attribute table. Asking for one gets you a short summary instead, and the number that matters is on the last line. This is a 15,000 rpm Seagate in the Camp's backplane:
professorlinux@L:~# sudo smartctl -i /dev/sdm
Vendor: SEAGATE
Product: ST9300653SS
Rotation Rate: 15000 rpm
Serial number: 6XN1S5J0
Transport protocol: SAS (SPL-3)
professorlinux@L:~# sudo smartctl -H /dev/sdm
SMART Health Status: OK
professorlinux@L:~# sudo smartctl -A /dev/sdm
Current Drive Temperature: 29 C
Accumulated power on time, hours:minutes 7249:47
Manufactured in week 33 of year 2012
Accumulated start-stop cycles: 83
Elements in grown defect list: 43Health status OK, and forty-three sectors reassigned since it left the factory. The grown defect list, the GLIST, is the SAS equivalent of attribute 5: sectors retired in the field. It sits alongside the PLIST, the defect list burned in at manufacture, which is normal. Only the grown list is news.
Two other SAS disks in the same backplane put that 43 in context:
professorlinux@L:~# sudo smartctl -A /dev/sdg
Accumulated power on time, hours:minutes 59138:31
Elements in grown defect list: 0
professorlinux@L:~# sudo smartctl -A /dev/sdf
Accumulated power on time, hours:minutes 33253:13
Elements in grown defect list: 1The clean disk has run for 59138 hours. The one with a single defect has run for 33253. The disk with 43 has run for 7249, the shortest service life of the three. Age is not the explanation, and that is the point of trending a counter instead of reading it once. On this host, sdm is the drive that gets replaced first.
One trap worth knowing. smartctl -l defects reads an ATA pending-defects log, so on a SAS device smartctl has nothing to do and politely bounces you:
professorlinux@L:~# sudo smartctl -l defects /dev/sdm
SCSI device successfully opened
Use 'smartctl -a' (or '-x') to print SMART (and more) informationTake the hint. On SAS, -a adds the SCSI error counter log, which smartctl(8) documents alongside a non-medium error count: errors that were not the media's fault, such as interface or servo problems. A grown defect list climbing while non-medium errors stay flat is a surface problem. Both climbing together points at the path to the drive.
| What the RAW values say | Call | Why |
|---|---|---|
5, 187, 188, 197, 198 all zero, -H PASSED | Keep. Run smartd and a periodic long test | The healthy baseline our Seagate shows |
| 5 small and unchanged across two readings a week apart, 197 and 198 zero | Watch, with a date on the calendar and a spare on the shelf | Backblaze calls nonzero a reason to investigate. Google measured over 14 times the failure odds within 60 days after a first reallocation |
| 197 nonzero, any amount | Act now. Back up, then run a long test | Unreadable data today. Google put a first pending count at 16 times the odds within 60 days |
| 187 nonzero | Pull | Backblaze replaces at greater than zero |
| 198 nonzero, worse if 197 is too | Pull | A surface scan confirmed it. Google put offline reallocations at over 21 times |
| Any of the five above your last reading | Pull | The trend is the signal, not the absolute number |
-H FAILED, or any row with WHEN_FAILED set | Pull today | The firmware is predicting its own failure |
| SAS: grown defect list larger than last month | Pull | Same signal, different dialect |
Those numbers come from two public data sets. Backblaze's 2016 analysis reported that 4.2 percent of their operational drives had one or more of those five stats above zero, against 76.7 percent of their failed drives. The multipliers are from Pinheiro, Weber and Barroso's FAST 2007 paper on more than 100,000 Google disks.
Now the honest half. That same Backblaze figure leaves 23.3 percent of their failures giving no warning on any of the five. The Google paper is blunter: over 56 percent of failed drives had no count in any of the four strongest SMART signals, and over 36 percent had zero counts on every SMART variable tracked. SMART is a one-way test. It can tell you a disk is dying. It cannot tell you a disk is fine.
smartctl -t long /dev/sdX starts a full surface read scan and returns immediately. The results are not in that command's output. They land in the self-test log, which is what -l selftest prints:
professorlinux@L:~# sudo smartctl -l selftest /dev/sdl
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 17 -
# 2 Extended offline Completed without error 00% 17 -
# 3 Short offline Completed without error 00% 16 -Read the LifeTime column, then read it again. The newest entry ran at power-on hour 17. This disk's Power_On_Hours attribute says 22980. Nobody has asked it to check itself since its first day in service, which is true of most disks in most racks, and is the cheapest thing on this page to fix.
The drive will tell you how long a test takes:
professorlinux@L:~# sudo smartctl -c /dev/sdl
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 50) minutes.Two minutes and fifty minutes on a 250 GB disk. Scale that for a modern multi-terabyte drive before you promise anyone a maintenance window. Tests run during normal operation and cost some performance. smartctl -X aborts one.
A failing test does not print an error. It stops at the first unreadable sector and records where:
Source: smartmontools Bad Block HOWTO (https://www.smartmontools.org/wiki/BadBlockHowto)
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 90% 217 0x016561e9Ninety percent of the surface was still untested when it gave up, at power-on hour 217, and the last column is the sector that stopped it. That LBA counts 512-byte sectors from zero, per smartctl(8). It is the address you hand to a recovery attempt, or to the forced-rewrite procedure in the smartmontools Bad Block HOWTO, which destroys that block's contents to make the drive fix or remap it.
A drive can fail this test and still report PASSED on -H. The smartmontools FAQ explains that this usually means there is an uncorrectable sector on the disk. The log wins.
Put DEVICESCAN -a -m [email protected] -s (S/../.././01|L/../../6/03) in /etc/smartd.conf and, per smartd.conf(5), smartd watches every disk it finds for health status, attribute changes, new error log entries and new self-test failures, mails you when any of them move, and runs a short test daily at 01:00 with a long test every Saturday at 03:00.
Redundancy lowers the bar for pulling a disk. It does not raise it. A rebuild or a resilver reads essentially everything on every surviving member, so a latent pending sector on a disk you decided to watch surfaces at the exact moment you have no redundancy left to survive it. That is the whole argument for scrubbing: a zpool scrub verifies every block's checksum and repairs damage from a replica while the array is still whole, and an md check reads every block for the same reason. Backblaze's tolerance for a nonzero counter exists inside massive redundancy. One copy of the data means pulling sooner than they do, not later. If the disk is NVMe, none of these attributes exist and the fields are different: see the NVMe smart-log article.
| Command or field | What it gives you |
|---|---|
smartctl -i /dev/sdX | Model, serial, sector size, whether SMART is enabled |
smartctl -H /dev/sdX | Firmware verdict. PASSED on ATA, SMART Health Status: OK on SAS |
smartctl -A /dev/sdX | The attribute table, or the SAS summary with the grown defect list |
smartctl -a /dev/sdX | The above plus the error and self-test logs |
smartctl -x /dev/sdX | -a plus the extended logs and more SCSI log pages |
smartctl -c /dev/sdX | This drive's own self-test time estimates |
smartctl -t short, -t long | Start a self-test. -X aborts a running one |
smartctl -l selftest /dev/sdX | Where self-test results actually live |
smartctl -l error /dev/sdX | Recent logged errors with power-on timestamps |
smartctl --scan | Every device smartctl sees, with the -d type it guessed |
| RAW_VALUE on 5, 197, 198 | Remapped, pending and uncorrectable sector counts |
Elements in grown defect list | The SAS equivalent of attribute 5 |
Verified on smartctl 7.2 2020-12-30 r5155, kernel 5.14.0-503.40.1.el9_5.x86_64, against SATA, SAS and SSD devices on the same bare-metal host.
Does a small, stable Reallocated_Sector_Ct mean the disk is dying? Not on its own, and not today. A handful of remapped sectors that has not moved between two readings a week apart is a watch item. But Google's disk study measured over 14 times the failure odds within 60 days after a first reallocation, so watching means re-reading the counter on a schedule with a replacement on the shelf, not noting it and moving on.
Current_Pending_Sector is 8. Can I clear it, and should I? You can. A successful read or any write to those sectors clears the count, either in place or by remapping to spares, which pushes attribute 5 up instead. Eight is usually one 4096-byte physical sector on a 512e drive, so it is one defect, not eight. Back up first, and treat a pending count that returns after a rewrite as a retirement notice.
Reallocated_Event_Count or Reallocated_Sector_Ct, which do I trust? Attribute 5 counts sectors, 196 counts remap operations, and one operation can carry more than one sector, so the two legitimately differ. Plenty of drives omit 196 entirely, including the Seagate captured above. Judge by 5.
smartctl -H says PASSED but the long test keeps failing. Which wins? The test. PASSED only means the firmware's own thresholds have not tripped, and our captured Seagate prints those thresholds as ---, meaning it publishes none. The smartmontools FAQ notes that failing a self-test while still reporting PASSED usually means there is an uncorrectable sector on the disk. Trust the self-test log and the raw counters over the summary verdict.
More from the Field Manual, or work through the training tracks.