Learn › BMC & Server Management › Operations
Something happened at 3 a.m. Read the server's black box: decode SEL severities with interactive diagrams, pull the 03:47 crash timeline from a live BMC, export the evidence, and clear the log the way ops teams do it.
Your phone lit up at 04:15. The NOC feed: bay14-r750xs, the Dell PowerEdge R750xs in Bay 14, Row 7 at the DC-EAST facility in Ashburn, dropped off monitoring at 03:47 and was back online by 04:05 like nothing happened. Nobody was on the floor. Nobody touched it. Eighteen minutes of silence, then business as usual.
The NOC will not return this box to the production pool until someone explains the gap. That someone is you. Work Order DC-EAST-WO-1006 has four verbs on it: read the event log, decode it, export it, and clear it properly.
Here is what makes the job possible: the server kept a black box recording of its own death. Every server does. This lesson is about finding that recording, reading it like an investigator, and handling it like evidence.
This is a Practice Zone. The diagrams below react to you: click the boxes, flip the switches. The practice terminals check only the one command being taught. The real machine comes at the end, under Ready to practice.
The BMC you met in First Contact does more than answer power questions. It keeps a diary, the System Event Log, or SEL. Every time a sensor crosses a threshold, the BMC writes a stamped entry: when it happened, which sensor spoke, what it said, and how bad it was. It never asks the host OS for permission, and it does not need the host alive to do it.
That last part is the whole story tonight. The BMC runs on standby power. When the host crashed at 03:47, the recorder kept recording. Flip the host below and watch which lane stays lit.
{ "height": 440, "caption": "How a 3 a.m. event becomes a log entry. Click any box, then crash the host.", "nodes": [ { "id": "cpu0", "label": "CPU0 temp", "kind": "sensor", "x": 250, "y": 30, "detail": "A thermal sensor on the first processor. When its reading crosses a threshold, it reports the crossing to the BMC." }, { "id": "fan3", "label": "FAN3 tach", "kind": "sensor", "x": 250, "y": 130, "detail": "The tachometer on fan 3 reports rotation speed. A fan slowing down is an event worth recording." }, { "id": "psu", "label": "Power unit", "kind": "psu", "x": 250, "y": 230, "detail": "The power supplies report AC loss the instant the feed drops, in the last milliseconds before everything goes dark." }, { "id": "you", "label": "You (NOC)", "kind": "admin", "x": 0, "y": 330, "detail": "Reading the log after the fact, from your desk. The recorder wrote everything down so you would not have to be there at 3 a.m." }, { "id": "bmc", "label": "BMC", "kind": "bmc", "x": 540, "y": 130, "detail": "The always-on controller. It runs on standby power, so it keeps recording while the host is dead." }, { "id": "store", "label": "Event store", "kind": "sensor", "x": 830, "y": 40, "detail": "Every event lands here with a timestamp, a sensor name, a description, and a severity. This is the black box." }, { "id": "host", "label": "Host OS", "kind": "host", "x": 830, "y": 280, "detail": "The big machine, the thing that crashed at 03:47. It cannot testify about its own death. The BMC can." } ], "edges": [ { "from": "cpu0", "to": "bmc", "label": "threshold crossed", "kind": "plain" }, { "from": "fan3", "to": "bmc", "label": "speed too low", "kind": "plain" }, { "from": "psu", "to": "bmc", "label": "AC lost", "kind": "power" }, { "from": "bmc", "to": "store", "label": "append entry", "kind": "plain" }, { "from": "you", "to": "bmc", "label": "read anytime", "kind": "oob" }, { "from": "bmc", "to": "host", "label": "watches", "kind": "plain" } ], "toggle": { "label": "Host at 03:47:", "on": "Running", "off": "Crashed", "dimOff": ["host", "e:bmc-host"] } }
Crash the host and look again: the sensors-to-BMC-to-store lane never dims. The diary keeps writing while the patient is flatlined. That is why you can investigate a 3 a.m. crash at 8 a.m. without a single person having seen it happen.
>>> Kept recording. The recorder is out-of-band like everything else about the BMC: its own processor, its own power, its own storage. That is exactly why it can testify about a crash that killed the OS. The patient flatlined; the diary kept writing.
Fourteen entries are waiting for you tonight, and they are not all equally loud. Every entry carries a severity:
IPMI names the fences with its own vocabulary, and the vocabulary trips people up. Upper Non-Critical is the WARNING fence: getting warm, not yet dangerous. Upper Critical is the critical fence: too hot, act now. Fans use the Lower versions, because a fan fails by slowing down, crossing fences on the way DOWN.
>>> Warning. Non-Critical is the formal IPMI name of the warning threshold, so a Non-Critical crossing is a Warning severity event. Misreading Non-Critical as critical is the classic rookie mistake, and tonight it matters: FAN3 crossed this warning fence just 3 seconds before it crossed the real Critical one.
Severity is also a filter. A 14-row log reads as noise until you keep only the loud rows, and then it reads as a story. Here is tonight's incident as the log recorded it, cause to effect. Flip the view.
{ "height": 340, "caption": "One incident, six beats. Follow the arrows, then filter to Critical only.", "nodes": [ { "id": "inlet", "label": "02:14 Inlet warm", "kind": "sensor", "x": 0, "y": 40, "detail": "Inlet_Temp crosses its warning fence: Upper Non-Critical going high. Severity: Warning. The cooling is already losing the fight, and nobody knows yet." }, { "id": "cpus", "label": "03:42 CPUs critical", "kind": "sensor", "x": 320, "y": 40, "detail": "CPU0 then CPU1 cross Upper Critical within three seconds of each other. Severity: Critical. Both processors are over the fence." }, { "id": "fan3", "label": "03:45 FAN3 failing", "kind": "sensor", "x": 640, "y": 40, "detail": "FAN3 crosses Lower Non-Critical, then Lower Critical three seconds later. A fan giving out while the CPUs cook is the worst possible timing." }, { "id": "oscrash", "label": "03:47:01 OS stop", "kind": "host", "x": 640, "y": 240, "detail": "OS runtime critical stop. Severity: Critical. This is the exact moment the NOC monitoring went dark." }, { "id": "acloss", "label": "03:47:05 AC lost", "kind": "psu", "x": 320, "y": 240, "detail": "System power down, Loss of AC. Four seconds after the OS died, the power feed itself dropped." }, { "id": "recovery", "label": "04:02 AC restored", "kind": "host", "x": 0, "y": 240, "detail": "AC restored at 04:02:15, boot initiated 04:02:30, boot completed 04:05:44. The server walked back in like nothing happened. The log knows better." } ], "edges": [ { "from": "inlet", "to": "cpus", "label": "88 min of warming", "kind": "plain" }, { "from": "cpus", "to": "fan3", "label": "3 min", "kind": "plain" }, { "from": "fan3", "to": "oscrash", "label": "90 sec", "kind": "plain" }, { "from": "oscrash", "to": "acloss", "label": "4 sec", "kind": "plain" }, { "from": "acloss", "to": "recovery", "label": "15 min dark", "kind": "power" } ], "toggle": { "label": "View:", "on": "Full log", "off": "Critical only", "dimOff": ["inlet", "recovery", "e:inlet-cpus", "e:acloss-recovery"] } }
Read it backwards from the crash: the OS did not just stop at 03:47. The room warmed for 88 minutes, both CPUs went critical, a fan gave out, and THEN the OS died, four seconds before the power feed dropped. The 03:47 page was the last line of a story that started at 02:14. Severity filtering is how you find that story inside a wall of rows.
Time to read the real thing. The BMC's Dell-style shell is one command away, the same door you opened in First Contact: sshpass -p ADMIN ssh -o StrictHostKeyChecking=no admin@$BMC_IP lands you at the /admin1-> prompt.
Inside that shell, the vendor command for the event log is one word: get, plus the log's three-letter name. Run it.
prompt: /admin1-> answer: getsel output: System Event Log ============================================================
ID Timestamp Sensor Event Severity -------------------------------------------------------------------------------------- 1 06/09/2026 22:00:05 System_Power System boot completed Informational 2 06/10/2026 02:14:30 Inlet_Temp Upper Non-Critical going high Warning 3 06/10/2026 02:18:45 CPU0_Temp Upper Non-Critical going high Warning 4 06/10/2026 02:22:12 CPU1_Temp Upper Non-Critical going high Warning 5 06/10/2026 03:42:15 CPU0_Temp Upper Critical going high Critical 6 06/10/2026 03:42:18 CPU1_Temp Upper Critical going high Critical 7 06/10/2026 03:42:20 Exhaust_Temp Upper Non-Critical going high Warning 8 06/10/2026 03:45:30 FAN3 Lower Non-Critical going low Warning 9 06/10/2026 03:45:33 FAN3 Lower Critical going low Critical 10 06/10/2026 03:47:01 System_Power OS runtime critical stop Critical 11 06/10/2026 03:47:05 System_Power System power down - Loss of AC Critical 12 06/10/2026 04:02:15 System_Power Power unit AC restored Informational 13 06/10/2026 04:02:30 System_Power System boot initiated Informational 14 06/10/2026 04:05:44 System_Power System boot completed Informational
Total entries: 14 hint: The vendor shell verb for reading the SEL: get plus the log's three-letter acronym, as one word.
Five columns: an ID, a timestamp, the sensor that spoke, what it said, and the severity you just learned to decode. Find the cluster: IDs 10 and 11 are the death itself, the OS stop at 03:47:01 and the AC loss four seconds later. IDs 12 to 14 are the quiet recovery while you were still asleep. Type exit when you are done and you are back on your workstation.
The NOC cannot close an incident on your say-so. They want the log ON FILE, attached to the work order. Here is the catch: the BMC shell has no filesystem for you and no > redirect. Evidence has to land on YOUR workstation.
There is a second catch. This BMC's SSH door ignores a command tacked onto the ssh line: the shell swallows it and opens its normal session as if you had said nothing. What it WILL do is read commands from standard input. So you feed the command in with a pipe from your side of the wire:
echo getsel | sshpass -p ADMIN ssh -o StrictHostKeyChecking=no admin@$BMC_IP > ~/sel-evidence.txtRead it left to right: echo getsel writes the word getsel, the pipe feeds it into the SSH session as if you had typed it at the /admin1-> prompt, the table comes back on standard output, and YOUR shell, which does have a filesystem, owns the redirect. The whole log lands in ~/sel-evidence.txt with no human sitting in the session.
Once the evidence is a local file, your whole Linux toolbox applies to it. Pull out the loud rows with grep -i critical ~/sel-evidence.txt:
prompt: ops@dc-east-ws01:~$ answer: grep -i critical ~/sel-evidence.txt output: 2 06/10/2026 02:14:30 Inlet_Temp Upper Non-Critical going high Warning 3 06/10/2026 02:18:45 CPU0_Temp Upper Non-Critical going high Warning 4 06/10/2026 02:22:12 CPU1_Temp Upper Non-Critical going high Warning 5 06/10/2026 03:42:15 CPU0_Temp Upper Critical going high Critical 6 06/10/2026 03:42:18 CPU1_Temp Upper Critical going high Critical 7 06/10/2026 03:42:20 Exhaust_Temp Upper Non-Critical going high Warning 8 06/10/2026 03:45:30 FAN3 Lower Non-Critical going low Warning 9 06/10/2026 03:45:33 FAN3 Lower Critical going low Critical 10 06/10/2026 03:47:01 System_Power OS runtime critical stop Critical 11 06/10/2026 03:47:05 System_Power System power down - Loss of AC Critical hint: The filter that keeps matching lines, its ignore-case flag, the word critical, then the evidence file.
Count them: ten rows, not five. The filter matched the five true Critical rows AND five Warning rows, because Warning rows carry the threshold name Non-Critical. That is the trap from the severity ladder, live: grep matched the NAME of the fence, not the severity column. A good tech does not throw those passengers out. The warnings ARE the story of how the machine got to critical.
Now the twist that separates people who understand BMCs from people who memorize commands. Point the standards tool at the same controller and size ITS event log:
prompt: ops@dc-east-ws01:~$ answer: ipmitool -I lanplus -C 3 -H $BMC_IP -U ADMIN -P ADMIN sel info output: SEL Information Version : 1.5 (v1.5, v2 compliant) Entries : 0 Free Space : 16000 bytes Percent Used : 0% Last Add Time : Not Available Last Del Time : Not Available Overflow : false Supported Cmds : 'Delete' 'Reserve' hint: The remote IPMI form you always use, cipher suite and all, then two words: sel info.
Entries: 0. The wire answered instantly, the store is healthy with 16000 bytes free, and it is EMPTY. But you just read fourteen entries with the vendor command. Both are true, because this controller keeps two record stores behind two doors.
{ "height": 420, "caption": "Two doors, two record stores, one controller. Flip the door.", "nodes": [ { "id": "you", "label": "You (ops)", "kind": "admin", "x": 0, "y": 200, "detail": "Your workstation dc-east-ws01. Both doors are one command away." }, { "id": "sensors", "label": "13 sensors", "kind": "sensor", "x": 560, "y": 20, "detail": "Temperatures, fans, power draw. Every threshold crossing they report becomes an event." }, { "id": "sshdoor", "label": "tcp/22 SSH", "kind": "nic", "x": 280, "y": 80, "detail": "The vendor shell door. ssh admin@$BMC_IP lands you at the /admin1-> prompt." }, { "id": "ipmidoor", "label": "udp/623 IPMI", "kind": "nic", "x": 280, "y": 320, "detail": "The standards door. ipmitool -I lanplus speaks IPMI 2.0 to it, the same words against any vendor's BMC." }, { "id": "bmc", "label": "BMC", "kind": "bmc", "x": 560, "y": 200, "detail": "One controller, two record stores. Which log you see depends on which door you knock on." }, { "id": "vlog", "label": "Vendor event store", "kind": "sensor", "x": 850, "y": 80, "detail": "The rich vendor log: full timestamps, sensor names, severities. getsel reads it, clearsel wipes it." }, { "id": "selstore", "label": "Standards SEL chip", "kind": "sensor", "x": 850, "y": 320, "detail": "The terse, portable IPMI store. sel info sizes it, sel list reads it. On this controller it starts empty." } ], "edges": [ { "from": "you", "to": "sshdoor", "label": "ssh admin@", "kind": "mgmt" }, { "from": "sshdoor", "to": "bmc", "kind": "mgmt" }, { "from": "you", "to": "ipmidoor", "label": "ipmitool -I lanplus", "kind": "oob" }, { "from": "ipmidoor", "to": "bmc", "kind": "oob" }, { "from": "bmc", "to": "vlog", "label": "rich entries", "kind": "plain" }, { "from": "bmc", "to": "selstore", "label": "terse entries", "kind": "plain" }, { "from": "sensors", "to": "bmc", "label": "events", "kind": "plain" } ], "toggle": { "label": "Door:", "on": "Vendor shell (SSH)", "off": "Standard IPMI", "dimOn": ["ipmidoor", "selstore", "e:you-ipmidoor", "e:ipmidoor-bmc", "e:bmc-selstore"], "dimOff": ["sshdoor", "vlog", "e:you-sshdoor", "e:sshdoor-bmc", "e:bmc-vlog"] } }
Real Dell hardware makes the same split: an iDRAC keeps a terse standards SEL for ipmitool and a rich Lifecycle Log for its own tooling, and the rich one is where the detail lives. Knowing which door holds which log is the difference between "the SEL is empty, must be fine" and actually finding the 03:47 story.
The BMC in this lab is an emulator that plays this split honestly, with simplifications to know about: on a real iDRAC the vendor commands are racadm getsel and racadm clrsel (our shell drops the racadm prefix), the standards SEL would carry the incident too instead of starting empty, and a real iDRAC also serves its SEL over Redfish LogServices, which this emulator does not implement. The commands you type here are the real commands.
One gotcha worth knowing before night shift: sel list on an empty SEL prints SEL has no entries to standard error, not standard output. Redirect it into a file and the file comes out empty. An empty redirect file means an empty SEL, not a broken command.
Run sel time get against a fresh BMC and you may get a date in January 1970. That is not damage, it is an unset SEL clock counting from the Unix epoch. Real fleets genuinely see 1970-stamped entries after a BMC battery pull or clock reset. A timestamp is only as trustworthy as the clock behind it.
The last verb on the work order is the one with teeth. The SEL has fixed space, you saw it yourself: 16000 bytes. When it fills, new events get dropped, and the event you lose is always the one you needed. So ops teams run a discipline: document first, clear second. Export the log, attach it to the incident, and only then wipe it so the next incident starts on a clean page.
In the vendor shell, the wipe is clearsel: clear plus the log's name, one word, no are-you-sure. Read the log back afterwards and this controller reports No SEL entries recorded. A real iDRAC leaves one fresh Log cleared entry behind as a receipt; the discipline is identical either way.
>>> Gone forever. There is no undo, no archive, no recycle bin on a management controller. That is why the NOC cannot close an incident without the export attached, and why document-first-clear-second is a discipline, not a suggestion.
Time to work the incident for real. On the machine you are about to open, the BMC beside it holds the 03:47 log. The address is preset in $BMC_IP, credentials ADMIN / ADMIN. Six objectives close Work Order DC-EAST-WO-1006:
1. Open the black box. Get inside the BMC's own shell and read the event log from the inside. 2. Size the standards SEL. Over the wire, no SSH session: entry count, free space, overflow. 3. List the standards SEL. Same door, full listing. Report what it tells you, even if it is nothing. 4. Export the evidence. The full vendor log, saved to ~/sel-evidence.txt, without opening an interactive session. 5. Isolate the critical trail. Just the rows that tell the 03:47 story, in ~/sel-critical.txt. 6. The challenge. Close the incident: clear the log properly and leave proof of the cl
Practice Event Log in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.