Learn › BMC & Server Management › First Day on the Floor
A thermal alarm tripped in Bay 14 overnight. Learn how sensors, thresholds, and the BMC cooling loop work with interactive diagrams, then audit a live Dell BMC: read all 13 sensors, decode alarm thresholds over raw IPMI, and name the sensor that is out of spec.
It is 03:12 at the DC-EAST facility in Ashburn, Virginia, and your phone is buzzing. The NOC monitoring board has flagged a thermal alarm on the Dell PowerEdge R750xs in Bay 14, Row 7. Two CRAC units in that row went down for maintenance yesterday afternoon, and the room has been warming up ever since.
Work Order DC-EAST-WO-1004 is short: read the temperatures, fans, and power draw from the BMC, and name the sensor that is out of spec before dayshift decides whether Bay 14 stays put or its workloads move.
Here is the puzzle. Nothing has crashed. The server is up, workloads are running, users are asleep and happy. There is no error in any application log, because applications do not feel heat. So where do you even look?
You already know the answer from your first day: the BMC. That little always-on computer has been taking the machine's temperature all night, literally. Thirteen sensors, sampled continuously, compared against alarm thresholds, every crossing logged with a timestamp. Tonight you learn to read them.
This is a Practice Zone. The diagrams below react to you: click the boxes, flip the switches. The real machine comes at the end, under Ready to practice.
Before you read a single sensor, you need to know where the sensors ARE. A rack server is a wind tunnel: cold air in the front, hot air out the back, and every sensor sits at a station along that airflow.
The CRAC unit (Computer Room Air Conditioning) pushes cold air under the raised floor and up through perforated tiles in front of the rack. That air enters the server at the front bezel, crosses two hot CPUs, and leaves through the back, dragged the whole way by six counter-rotating fans. The BMC polls a thermometer at every stage.
{ "height": 400, "caption": "Follow the air, front to back. Click any station, then fail the CRAC.", "nodes": [ { "id": "crac", "label": "CRAC unit", "kind": "net", "x": 0, "y": 200, "detail": "Computer Room Air Conditioning. Pushes cold air under the raised floor and up through perforated tiles in front of the rack. Two of Row 7's units went down for maintenance yesterday afternoon." }, { "id": "inlet", "label": "Inlet_Temp", "kind": "sensor", "x": 220, "y": 200, "detail": "The room's thermometer, mounted where cold air enters the chassis. First to know when facility cooling fails." }, { "id": "cpu0", "label": "CPU0_Temp", "kind": "sensor", "x": 440, "y": 90, "detail": "Silicon temperature of socket 0. Measures the work the server is doing, not the room it is in." }, { "id": "cpu1", "label": "CPU1_Temp", "kind": "sensor", "x": 440, "y": 310, "detail": "Socket 1. On a healthy machine the two CPU temperatures track each other within a few degrees." }, { "id": "exhaust", "label": "Exhaust_Temp", "kind": "sensor", "x": 660, "y": 200, "detail": "The receipt: inlet heat plus CPU heat, measured on the way out the back of the chassis." }, { "id": "fans", "label": "FAN1-6", "kind": "sensor", "x": 660, "y": 0, "detail": "Six counter-rotating fans the BMC commands. They drag the whole wind tunnel through the chassis." }, { "id": "bmc", "label": "BMC", "kind": "bmc", "x": 880, "y": 90, "detail": "Polls all 13 sensors continuously on standby power. It was watching all night while everyone slept." }, { "id": "noc", "label": "You (NOC)", "kind": "admin", "x": 880, "y": 310, "detail": "The 03:12 page came from the BMC crossing a threshold, not from the OS. Applications do not feel heat." } ], "edges": [ { "from": "crac", "to": "inlet", "label": "cold air", "kind": "plain" }, { "from": "inlet", "to": "cpu0", "kind": "plain" }, { "from": "inlet", "to": "cpu1", "kind": "plain" }, { "from": "cpu0", "to": "exhaust", "label": "hot air", "kind": "plain" }, { "from": "cpu1", "to": "exhaust", "kind": "plain" }, { "from": "bmc", "to": "inlet", "label": "polls", "kind": "mgmt" }, { "from": "bmc", "to": "cpu0", "kind": "mgmt" }, { "from": "bmc", "to": "exhaust", "kind": "mgmt" }, { "from": "bmc", "to": "fans", "kind": "mgmt" }, { "from": "bmc", "to": "noc", "label": "alert", "kind": "oob" } ], "toggle": { "label": "CRAC unit:", "on": "Running", "off": "Down for maintenance", "dimOff": ["crac", "e:crac-inlet"] } }
Click through the stations, then flip the CRAC switch and watch the cold-air feed die. The geography is the lesson:
When the cold air stops, the inlet warms FIRST and everything downstream follows. The room is part of the machine.
The inlet reads 36 degrees. Is that bad? You cannot know from the number alone. A reading becomes an alarm only when it crosses a threshold, and every sensor carries its own set, stored inside the BMC.
For rising temperatures, IPMI defines three upper rungs, each more serious than the last:
For this machine's inlet sensor the rungs sit at 38, 42, and 47 degrees C. Flip the toggle between a quiet night and tonight's overnight peak and watch which rungs fire.
{ "height": 420, "caption": "Three rungs between a warm room and a dead server. Flip the night.", "nodes": [ { "id": "reading", "label": "Inlet_Temp reading", "kind": "sensor", "x": 0, "y": 170, "detail": "Just a number until the BMC compares it with something. Quiet nights it sits near 24 C. Tonight it peaked at 39." }, { "id": "bmc", "label": "BMC compare loop", "kind": "bmc", "x": 280, "y": 170, "detail": "Checks every reading against three upper rungs. For this sensor: warn at 38, critical at 42, non-recoverable at 47." }, { "id": "sel", "label": "SEL entry", "kind": "net", "x": 620, "y": 0, "detail": "Every crossing is logged with a timestamp in the System Event Log. This paper trail is how you will reconstruct tonight." }, { "id": "noc", "label": "NOC alert", "kind": "admin", "x": 620, "y": 130, "detail": "The page that buzzed you at 03:12 started at this rung. A warning exists so a human can fix the room in time." }, { "id": "throttle", "label": "Throttle host", "kind": "host", "x": 620, "y": 260, "detail": "At the critical rung the BMC slows the CPUs to shed heat. Performance drops so the hardware survives." }, { "id": "kill", "label": "Emergency power-off", "kind": "psu", "x": 620, "y": 390, "detail": "Non-recoverable means damage territory. The BMC will kill power rather than cook the board." } ], "edges": [ { "from": "reading", "to": "bmc", "label": "sampled", "kind": "mgmt" }, { "from": "bmc", "to": "sel", "label": "crosses 38 (UNC)", "kind": "oob" }, { "from": "bmc", "to": "noc", "label": "crosses 38 (UNC)", "kind": "oob" }, { "from": "bmc", "to": "throttle", "label": "crosses 42 (UC)", "kind": "power" }, { "from": "bmc", "to": "kill", "label": "crosses 47 (UNR)", "kind": "power" } ], "toggle": { "label": "Inlet tonight:", "on": "24 C, quiet night", "off": "39 C overnight peak", "dimOn": ["e:bmc-sel", "e:bmc-noc", "e:bmc-throttle", "e:bmc-kill"], "dimOff": ["e:bmc-throttle", "e:bmc-kill", "throttle", "kill"] } }
On the alarm night, only the warning rung fired. The rungs are independent, and each one buys the next one time: a warning at 38 exists so a human can fix the room before the machine has to defend itself at 42.
>>> The inlet. Alarm math is distance to threshold, not size of the number: CPU0 has 13 degrees of margin, the inlet has 2. And the inlet measures the one thing the server cannot fix about itself, the room. The BMC can spin fans to save a hot CPU; it cannot air-condition Ashburn.
The BMC is not a passive gauge. It runs a control loop: read the temperatures, command the fans, read again. When the room started warming yesterday afternoon, the BMC did not just log it. It spun six fans from a quiet 8500 RPM to over 12000 and held them there all night.
That response shows up somewhere people forget to look: the power draw. Fans at full tilt plus hot silicon pulling harder means System_Power jumped from roughly 500 W to 835 W. Cooling is never free; the electric bill pays for it.
{ "height": 380, "caption": "The loop that kept Bay 14 alive tonight. Flip the night.", "nodes": [ { "id": "cpus", "label": "CPUs, 72 C", "kind": "host", "x": 0, "y": 80, "detail": "Workload heat. The server kept serving all night, and that heat had to go somewhere." }, { "id": "sensors", "label": "13 sensors", "kind": "sensor", "x": 260, "y": 0, "detail": "Four temperatures, six fan tachometers, three power meters, sampled continuously by the BMC." }, { "id": "bmc", "label": "BMC fan control", "kind": "bmc", "x": 520, "y": 80, "detail": "Reads temperatures, decides fan duty, reads again. No OS involved: this loop runs even while a machine boots." }, { "id": "fans", "label": "FAN1-6", "kind": "sensor", "x": 260, "y": 200, "detail": "Quiet night: about 8500 RPM. Alarm night: 12100 to 12600 RPM, status still OK. Loud is the response, not the fault." }, { "id": "psu", "label": "System_Power", "kind": "psu", "x": 0, "y": 300, "detail": "Quiet night: about 500 W. Alarm night: 835 W. Fans at full tilt and hot silicon both pull more current. Cooling is never free." }, { "id": "noc", "label": "NOC", "kind": "admin", "x": 780, "y": 80, "detail": "Hears about the fight only when a threshold is crossed. The loop itself runs in silence." } ], "edges": [ { "from": "cpus", "to": "sensors", "label": "heat", "kind": "plain" }, { "from": "sensors", "to": "bmc", "label": "readings", "kind": "mgmt" }, { "from": "bmc", "to": "fans", "label": "spin up", "kind": "mgmt" }, { "from": "fans", "to": "cpus", "label": "airflow", "kind": "plain" }, { "from": "fans", "to": "psu", "label": "draw", "kind": "power" }, { "from": "bmc", "to": "noc", "label": "warning", "kind": "oob" } ], "toggle": { "label": "Night:", "on": "Quiet", "off": "Thermal alarm", "dimOn": ["e:bmc-noc"], "dimOff": [] } }
Flip the night and read the node details: the loop runs either way, quiet or alarmed. What changes is how hard it works, what it costs in watts, and whether it has to tell anyone.
>>> The fans are doing exactly what the BMC told them to do, which is why their status is OK at full speed. On the floor, loud is usually the response, not the fault. When you hear a server screaming, ask what it is reacting to before you blame the thing making the noise.
Time to see the real table. You know the way in from your first work order: SSH to the BMC's Dell shell at $BMC_IP, credentials ADMIN / ADMIN. Inside, at the /admin1-> prompt, one command reads every sensor at once: getsensorinfo.
prompt: /admin1-> answer: getsensorinfo output: Sensor Readings ============================================================
Temperature Sensors ------------------------------------------------------------ Sensor Value Unit Status ---------------------------------------- CPU0_Temp 72 degrees C NC CPU1_Temp 68 degrees C NC Inlet_Temp 36 degrees C NC Exhaust_Temp 58 degrees C NC
Fan Sensors ------------------------------------------------------------ Sensor Value Unit Status ----------------------------- FAN1 12400 RPM OK FAN2 12200 RPM OK FAN3 12600 RPM OK FAN4 12300 RPM OK FAN5 12100 RPM OK FAN6 12500 RPM OK
Power Sensors ------------------------------------------------------------ Sensor Value Unit Status ------------------------------------ PSU1_Power 420 Watts OK PSU2_Power 415 Watts OK System_Power 835 Watts OK
Summary: 13 sensors total, 9 OK, 4 Warning, 0 Critical hint: It is one word, get sensor info, all lowercase, typed at the /admin1-> prompt.
Read it like an operator:
One more ops move. The NOC wants evidence files, not screenshots, and you do not need an interactive session to make one. The BMC shell reads commands from standard input, so from your workstation you can fire one command through a pipe and capture everything it prints:
echo getsensorinfo | sshpass -p ADMIN ssh -o StrictHostKeyChecking=no admin@$BMC_IP > ~/sensor-audit.txtThe login banner and prompts land in the file along with the table; that is fine, evidence has to be complete, not pretty. And once the table is in a file, grep carves out whatever slice a report needs, like grep FAN ~/sensor-audit.txt for just the six fan rows.
This BMC is a standards-faithful emulator: it speaks real IPMI on the wire and a Dell-style shell, but its readings are set by the scenario rather than physics, it has no voltage rail sensors (a production iDRAC also reports 12V, 5V, 3.3V, and VCORE rails), and it does not publish a sensor catalog over IPMI, so naming sensors happens in this shell rather than via ipmitool sdr. The commands and the reflexes are the real thing.
Where do the three rungs actually live? Inside the BMC itself, and any IPMI client can ask for them byte by byte. The raw subcommand of ipmitool sends a hand-built IPMI request: network function 0x04 (the sensor and event space), command 0x27 (Get Sensor Thresholds), sensor number 0x03 (the inlet). Same remote form as always, cipher suite and all.
prompt: ops@dc-east-ws01:~$ answer: ipmitool -I lanplus -C 3 -H $BMC_IP -U ADMIN -P ADMIN raw 0x04 0x27 0x03 output: 38 00 00 00 26 2a 2f hint: Type it exactly: ipmitool -I lanplus -C 3 -H $BMC_IP -U ADMIN -P ADMIN raw 0x04 0x27 0x03
Seven bytes back. Decode them left to right:
38 is the readable mask, binary 00111000: the three upper thresholds exist and are readable, the three lower ones are not set.00 00 00 are the unused lower rungs.26 2a 2f are the upper rungs in hex: 0x26 = 38 (warn), 0x2a = 42 (critical), 0x2f = 47 (non-recoverable).So the inlet is reading 36 against a warning rung of 38. Two degrees of headroom. Then why does the table still show a warning latched? Because of what happened yesterday, and the BMC never forgets. Inside the Dell shell, getsel prints the System Event Log, the paper trail from the threshold ladder:
System Event Log
============================================================
ID Timestamp Sensor Event Severity
-------------------------------------------------------------------------------------
1 06/10/2026 08:15:00 System_Power System boot completed Informational
2 06/10/2026 14:22:10 Inlet_Temp Upper Non-Critical going high Warning
3 06/10/2026 14:25:33 CPU0_Temp Upper Non-Critical going high Warning
4 06/10/2026 14:25:41 CPU1_Temp Upper Non-Critical going high Warning
Total entries: 4Read the trip ORDER. The inlet crossed its warning rung at 14:22:10. The CPUs followed three minutes later, at 14:25. The room warmed first and the silicon followed: the finger points at the failed CRAC units, not at the server. The reading has since settled just under the rung with the warning still latched, which is exactly how a real BMC behaves around a hovering threshold.
Two forward pointers, honestly given: this emulator's IPMI-side event log starts empty, so the Dell shell's getsel is where this scenario's timeline lives (full SEL forensics over IPMI comes in the Operations module). And Redfish, waiting in Module 3, publishes these same three rungs as clean JSON over HTTPS.
>>> 42. The critical rung is the second-to-last byte, 0x2a: two sixteens plus ten make 42. The 38 at the front is a trap twice over: it is the readable-mask byte, not a temperature, and the real warning rung of 38 degrees hides in hex as 0x26.
Time to do it for real. Work Order DC-EAST-WO-1004 closes when dayshift can read your evidence and act on it. The BMC address is preset in $BMC_IP, credentials ADMIN / ADMIN.
Six objectives:
1. Walk the sensor floor. Open the BMC's Dell shell and read all 13 sensors. Count the warnings. 2. File the audit. Capture the full sensor table to ~/sensor-audit.txt in one shot from the workstation, no interactive session. 3. Isolate the fan story. Only the six fan lines, in ~/fan-report.txt. 4. Ask the wire. Pull the inlet sensor's three alarm rungs over raw IPMI and decode the hex in your head. 5. Establish the timeline. Capture the BMC event log to ~/event-timeline.txt and note what tripped first. 6. The challenge. Dayshift wants ~/thermal-verdict.txt naming the out-of-spec sensor, with its reading. No hints on this one.
Here is the deal: the workspace shows no commands. It hands you one objective at a time, and you recall the move. That recall is the whole point; it is how tonight's reading becomes your skill.
Stuck is normal. Hit Request a signal on any objective for a nudge, or ask Daemon. It knows exactly which task you are on and will nudge, not spoil.
Your workstation dc-east-ws01 is booting in the bay below, with the Dell BMC beside it on the management network. $BMC_IP is preset, credentials ADMIN / ADMIN. Close Work Order DC-EAST-WO-1004 and tell dayshift whether Bay 14 stays or moves.
Practice Sensor Readings in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.