Learn › BMC & Server Management › Sensors and Thresholds
Final test: the whole Sensors and Thresholds chapter. Read the five live sensors three ways with sensor list, sdr list and sdr elist, open one in full with sensor get, read the six threshold columns, filter with sdr type, baseline the machine, name a fan failing at 600 RPM against a healthy 5400, watch the exhaust temperature really cross its 75 C critical line while the lab runs, read the events the controller raises by itself, correlate the fan with the heat, file the genuine Dell R730 reference, and state the verdict. 25 objectives on a real workstation called mgmt with a real IPMI 2.0 controller at 10.100.0.2 whose sensors move while you work.
This chapter covers the following topics:
sensor list, sdr list, sdr elistWhat you already know. Chapter 1 read sdr list, sensor list and sdr get on a host that had never been powered, where every sensor reported no reading and only the thresholds were readable. Chapter 3 read an event log in which a threshold crossing appeared as three records. This chapter reads the same commands against a powered host with a live fault, using software-emulated measurements, including a seeded reading that changes while you work.
Objectives. After this chapter you can:
sensor list, sdr list and sdr elist for a given questionok, ns, unc, ucr, unr, lnc, lcr, lnrThe machine. The software controller polls five emulated sensor values once a second, so the readings move while you work. The fault in this chapter is live: the exhaust temperature crosses its upper critical threshold partway through, and the controller logs the crossing itself. Boot takes about 3 minutes. This chapter is free; the machine is yours for 60 minutes and stops itself after 20 minutes idle.
This chapter uses a real Linux workstation and software-emulated IPMI hardware. The protocol requests and replies are real; controller identity, sensor values and host behavior may be seeded or emulated. Supplied physical-hardware captures are labeled separately.
These five questions establish whether you need this chapter in full. Each explanation names the section that covers the point.
>>> The lower thresholds come first, ascending, then the upper thresholds, also ascending. The order is fixed and identical for every sensor; na marks a threshold the sensor does not define. Section 4.2.
>>> Read the limits of the specific sensor. A high fan speed or low temperature can also be relevant on another platform. An unavailable threshold is not proof that every other monitoring path ignores that condition. Section 4.2.
>>> ns is no state. It is the correct answer on a powered-off host, where the sensors are not being driven. On a running host it is a finding: either the sensor has failed or the component it measures is absent. Section 4.3.
>>> The seeded scenario is designed around reduced cooling. Low fan speed and increasing exhaust support that explanation. They do not alone prove airflow volume or exclude every workload, sensor or facility issue. Section 4.7.
>>> Sensors move. Without a recorded starting point, "it is getting hotter" is an impression. With one, it is a measurement, and the difference decides whether a machine can wait for a maintenance window. Section 4.4.
Three commands read the same five sensors and print three different things. The first is sensor list, which prints the reading, the units, the state and the thresholds.
prompt: ops@mgmt:~$ answer: ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sensor list output: Inlet_Temp | 21.000 | degrees C | ok | na | na | na | 42.000 | 45.000 | 48.000 CPU0_Temp | 47.000 | degrees C | ok | na | na | na | 85.000 | 90.000 | 95.000 Exhaust_Temp | 68.000 | degrees C | nc | na | na | na | 65.000 | 75.000 | 80.000 Fan1_RPM | 600.000 | RPM | cr | 500.000 | 800.000 | 1000.000 | na | na | na PSU1_Power | 240.000 | Watts | ok | na | na | na | 800.000 | 896.000 | 960.000 hint: The sensor command has a list subcommand.
Your readings will differ, because the controller polls once a second and one of these sensors is moving. The thresholds will not differ.
Read the state column first, which is the fourth field. Three sensors read ok. Exhaust_Temp reads nc, and Fan1_RPM reads cr. Two sensors are outside their thresholds and the machine is telling you so in two characters.
Then read the numbers behind them. Exhaust_Temp is at 68 against an upper non-critical threshold of 65: over the warning line. Fan1_RPM is at 600 against a lower critical threshold of 800: below the act-now line, in a direction where lower is worse.
sdr list reads the same sensors from the sensor data repository and prints three columns.
prompt: ops@mgmt:~$ answer: ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sdr list output: Inlet_Temp | 21 degrees C | ok CPU0_Temp | 47 degrees C | ok Exhaust_Temp | 68 degrees C | nc Fan1_RPM | 600 RPM | cr PSU1_Power | 240 Watts | ok hint: sdr has a list subcommand.
Name, reading with its units, state. No thresholds.
This is the form to reach for when the question is "what is everything reading right now" rather than "how close is anything to a limit". On a chassis with two hundred sensors the difference in output length is the difference between a screen and a scroll.
sdr elist adds the sensor number and the entity the sensor is attached to.
prompt: ops@mgmt:~$ answer: ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sdr elist output: Inlet_Temp | 01h | ok | 55.1 | 21 degrees C CPU0_Temp | 02h | ok | 3.1 | 47 degrees C Exhaust_Temp | 03h | unc | 55.1 | 68 degrees C Fan1_RPM | 04h | lcr | 29.1 | 600 RPM PSU1_Power | 05h | ok | 10.1 | 240 Watts hint: The extended form of the sdr list subcommand.
Five columns: name, sensor number in hexadecimal, state, entity identifier, reading.
Two things here that the other forms do not give. The sensor number is what the event log prints, so Temperature #0x03 in a log is Exhaust_Temp here. The entity identifier says what the sensor is attached to: 55.1 is an air inlet or outlet, 3.1 is a processor, 29.1 is a fan, 10.1 is a power supply. On hardware with six fans, the entity identifier is what distinguishes them.
Note also that the state words differ from sdr list. Here Exhaust_Temp reads unc and Fan1_RPM reads lcr, naming the direction as well as the severity, where sdr list printed nc and cr.
Objective 5 asks for the distinction in writing, because choosing the wrong form is how a threshold gets missed.
prompt: ops@mgmt:~$ answer: printf 'sensor list and sdr list read the same sensors and print different things. sensor list prints the threshold table: the reading, the units, the state, then six threshold columns, lower non-recoverable through upper non-recoverable. sdr list prints the repository entry: name, reading and state, one line each. sdr elist adds the sensor id and the entity the sensor belongs to. Reach for sensor list when you need the thresholds, sdr elist when you need the identity.\n' > ~/sensors/list-vs-sdr.txt output: hint: printf with the explanation in single quotes and a trailing newline escape, redirected into the file.
The file reads:
sensor list and sdr list read the same sensors and print different things. sensor list prints the threshold table: the reading, the units, the state, then six threshold columns, lower non-recoverable through upper non-recoverable. sdr list prints the repository entry: name, reading and state, one line each. sdr elist adds the sensor id and the entity the sensor belongs to. Reach for sensor list when you need the thresholds, sdr elist when you need the identity.One sentence is the whole rule: sensor list for thresholds, sdr elist for identity. sdr list sits between them and is the quickest way to see everything at once.
sensor get takes a sensor name and prints its record with the live reading.
prompt: ops@mgmt:~$ answer: ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sensor get Exhaust_Temp output: Locating sensor record... Sensor ID : Exhaust_Temp (0x3) Entity ID : 55.1 Sensor Type (Threshold) : Temperature Sensor Reading : 68 (+/- 0) degrees C Status : Upper Non-Critical Lower Non-Recoverable : na Lower Critical : na Lower Non-Critical : na Upper Non-Critical : 65.000 Upper Critical : 75.000 Upper Non-Recoverable : 80.000 Positive Hysteresis : 2.000 Negative Hysteresis : 2.000 Assertions Enabled : unc- unc+ ucr- ucr+ unr- unr+ Deassertions Enabled : unc- unc+ ucr- ucr+ unr- unr+ hint: sensor get takes the sensor name exactly as sensor list prints it.
Your reading will differ and will be higher if you run this later.
Sensor Reading : 68 (+/- 0) degrees C. The bracketed figure is the accuracy the sensor record claims.
Status : Upper Non-Critical, the same fact as nc in the table, spelled out.
The six threshold lines are the table columns one per line, which is much easier to read correctly. The three lower thresholds are na: an exhaust that is too cold is not a fault. The three upper ones are 65, 75 and 80.
Hysteresis : 2.000 in both directions is the margin that stops a reading sitting on a threshold from logging an event on every poll. Having crossed 65, the reading must fall to 63 before the event deasserts.
Assertions Enabled : unc- unc+ ucr- ucr+ unr- unr+ lists the transitions the controller will log, + rising and - falling. All six upper transitions are enabled here, which is why the log in section 4.6 has the crossings in it.
Goal: read the six threshold columns without mixing up their severity. After the value, units and status, read the three lower limits followed by the three upper limits.
prompt: ops@mgmt:~$ answer: printf '%s\n' 'Column order: lower non-recoverable, lower critical, lower non-critical, upper non-critical, upper critical, upper non-recoverable.' 'In this lab Exhaust_Temp warns at 65 C, is critical at 75 C and non-recoverable at 80 C. Fan1_RPM warns below 1000 RPM, is critical below 800 RPM and non-recoverable below 500 RPM.' 'na means this threshold is unavailable or not applicable in this output. Check the SDR and platform policy before concluding that monitoring is absent.' > ~/sensors/thresholds.txt output: hint: Write the short evidence note with printf, then read it with cat ~/sensors/thresholds.txt.
Your file should record these statements:
Check: point to the warning and critical limit for each named sensor. Severity labels describe configured conditions; they do not by themselves prove permanent hardware damage.
Your goal: read the temperature sensors without searching through fans and power readings.
sdr type filters the sensor records by a named type. Keep the same connection options and add sdr type Temperature:
ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sdr type TemperatureTry it: at ops@mgmt:~$. Read each sensor's name, status and temperature. Compare the rows with the complete table you read earlier.
Check: the returned rows describe temperatures. You should be able to identify the warm exhaust reading and its status. Live values may change while you work.
For a multiword type, keep the words together with quotes: sdr type "Power Supply". If sdr type 0x08 reports that the type was not found, use the name. That message concerns the argument; it does not by itself mean the BMC address or password is wrong.
prompt: ops@mgmt:~$ answer: ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sdr type Temperature output: Inlet_Temp | 01h | ok | 55.1 | 21 degrees C CPU0_Temp | 02h | ok | 3.1 | 47 degrees C Exhaust_Temp | 03h | unc | 55.1 | 68 degrees C hint: sdr type takes the sensor type name, capitalised as the specification writes it.
This example returns Inlet_Temp, CPU0_Temp and Exhaust_Temp.
Read the row from left to right: name, sensor number, status, entity, reading. The example exhaust row ends in 68 degrees C and has status unc: it has crossed its upper non-critical threshold. The temperature and status can change on the live machine.
Fan, Voltage and Current are other common names. sdr type list shows names understood by ipmitool; it does not promise that this controller has a record for every type. An empty filtered result alone does not prove a physical part is missing.
Next: learn what the status words mean so a filtered list becomes useful evidence.
The state column is two or three characters and it is the fastest read on the whole table. Objective 15 records what they mean:
prompt: ops@mgmt:~$ answer: printf 'The state column, in the words ipmitool prints.\nok reports normal status for the conditions this command evaluates; it does not certify complete hardware health.\nns means no reading: the controller has nothing to report for that sensor.\nunc means upper non-critical, over the warning line.\nucr means upper critical, over the act now line.\nunr means upper non-recoverable, outside the safe range.\nlnc, lcr and lnr are the same three lines read from below, which is how a fan is judged.\nExhaust_Temp went from unc to ucr while this lab was running, and Fan1_RPM has read lcr since the controller first polled it.\n' > ~/sensors/status-words.txt output: hint: printf with one line per state word, separated by newline escapes, redirected into the file.
The file reads:
The state column, in the words ipmitool prints.
ok reports normal status for the conditions this command evaluates; it does not certify complete hardware health.
ns means no reading: the controller has nothing to report for that sensor.
unc means upper non-critical, over the warning line.
ucr means upper critical, over the act now line.
unr means upper non-recoverable, outside the safe range.
lnc, lcr and lnr are the same three lines read from below, which is how a fan is judged.
Exhaust_Temp went from unc to ucr while this lab was running, and Fan1_RPM has read lcr since the controller first polled it.Different ipmitool views use different abbreviations: sensor list may show nc or cr, while SDR output and threshold masks use unc/ucr or lnc/lcr to include direction. Use the row's value and limits to determine which direction applies.
For the direction-specific labels, the pattern is a direction letter and a severity: u or l, then nc, cr or nr. Once that is in mind the whole column reads at a glance, and a screen of two hundred sensors resolves to the two that are not ok.
The units column is not decoration. For analog sensors, the SDR describes how the raw reading is converted to engineering units. That conversion can include a multiplier, offset and exponents. The same raw value can therefore mean different things in different sensor records.
prompt: ops@mgmt:~$ answer: printf 'The units column says what the number means. Temperatures read degrees C, fans read RPM, and the supply reads Watts. For these analog readings, the SDR conversion maps raw 6 on Fan1_RPM to 600 RPM and raw 60 on PSU1_Power to 240 Watts. Other records can use offsets, exponents and nonlinear conversions. Never compare a number across two sensors without reading its units first.\n' > ~/sensors/units.txt output: hint: printf with the explanation in single quotes and a trailing newline escape, redirected into the file.
The file reads:
The units column says what the number means. Temperatures read degrees C, fans read RPM, and the supply reads Watts. For these analog readings, the SDR conversion maps raw 6 on FaPractice Sensors and Thresholds in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.