Learn › BMC & Server Management › Operations
This node reads healthy, but a job keeps dying. Watch a DIMM fail in real time across both planes: baseline DIMM.B2 as OK over Redfish, catch the correctable-error warning in the BMC SEL with getsel, watch the health flip to Critical live, and file the RMA evidence on a scenario that degrades while you look.
It is 02:10 at the DC-EAST facility in Ashburn, Virginia. A training job on the server in Bay 14 keeps dying and restarting. The on-call engineer paged you with one line: the node looks fine.
The monitoring dashboard agrees. Every light is green. CPU fine, power fine, memory fine. And yet the job will not stay up.
Here is the trap you are about to walk past. A machine that reads healthy right now is not the same as a machine that will stay healthy for the next ten minutes. Some parts fail slowly, and they announce it before they quit.
Work Order DC-EAST-WO-1042 is on your tablet: watch this node, catch the part that is starting to fail, and file the evidence before it takes the job down for good.
This is a Practice Zone. Type the commands in the boxes and read what comes back. The boxes are a safe sandbox that checks one command at a time. The real machine, the one that fails while you watch, comes at the end under Ready to practice.
This server answers to two management planes, and today you use both. Keep them straight, because they tell you different things.
Redfish is the modern management API (application programming interface). The BMC (baseboard management controller, the small computer that watches the big one) serves hardware facts as structured data over HTTP. Redfish reports the state of every component right now: healthy, warning, or critical.
The SEL (System Event Log) is the BMC black box recorder. It stamps every threshold crossing with the moment it happened and keeps the list on standby power. Redfish shows you the state now. The SEL shows you the story over time.
Most diagnostic work is archaeology: a part already failed, and you read back what happened. Today is different. Right now every component is healthy. You are not reading a crash from last night. You are on watch, and the failure has not finished happening yet.
Start on the Redfish plane, and write down what healthy looks like before anything moves. This BMC serves Redfish on port 5000. Every request carries HTTP Basic auth, the ADMIN / ADMIN credentials passed with the -u flag.
The tool curl is the command-line HTTP client: it asks, the BMC answers in JSON (structured text with named fields). The tool jq is the JSON field scalpel: name a path, get the value.
Read the health of one memory module, DIMM.B2, straight off its own page. The verdict lives at .Status.Health.
prompt: ops@dc-east-ws01:~$ answer: curl -s -u ADMIN:ADMIN http://$BMC_IP:5000/redfish/v1/Systems/1/Memory/DIMM.B2 | jq '.Status.Health' output: "OK" hint: curl -s -u ADMIN:ADMIN then the DIMM.B2 URL on port 5000, a padded pipe, then jq '.Status.Health'
OK. That is your baseline, on the record. Remember it, because the whole skill today is noticing when that one word changes.
Now cross to the other plane and read the recorder. The SEL is not on the Redfish port. You open the BMC own shell over SSH and read the log there. At the /admin1-> prompt, one word prints the whole log, oldest entry first: getsel.
The standard ipmitool sel list command does not surface these events on this controller, so you read the log through the BMC own shell with getsel. The two planes expose different things, and this is one of them.
prompt: /admin1-> answer: getsel output:
System Event Log ============================================================
ID Timestamp Sensor Event Severity ----------------------------------------------------------------------------- 1 07/30/2026 08:00:00 System_Power System boot completed Informational
Total entries: 1 hint: At the /admin1-> prompt, the one word that prints the System Event Log is getsel
One row. A clean boot at 08:00 and nothing since. The recorder has nothing to report. That is what quiet looks like, and it is about to get loud.
You now hold two readings of the same machine: Redfish says the module is OK right now, and the SEL shows one quiet boot. Before you go further, commit to an answer.
>>> The SEL. Redfish is a snapshot: this module is OK, or later, Critical. The SEL is the recording: correctable errors at one minute, persistent errors at two, a multi-bit failure at three. You need both, and you are about to watch both change.
Time is the tool now. Leave the log alone for a moment, then read it again. On a healthy machine nothing changes between two reads. On this one, something has.
The timestamp on a new row is the real moment it landed, in your own session clock, so your times will differ from the ones shown here. The boot row keeps its original 08:00 stamp. What matters is that a row appears that was not there before.
Run the same command again.
prompt: /admin1-> answer: getsel output:
System Event Log ============================================================
ID Timestamp Sensor Event Severity ---------------------------------------------------------------------------------------------------------- 1 07/30/2026 08:00:00 System_Power System boot completed Informational 2 07/30/2026 14:12:45 DIMM_B2 Correctable memory error rate exceeded for DIMM_B2 Warning
Total entries: 2 hint: The same one-word command as before: getsel
There it is. Row 2, stamped just now: DIMM_B2, Warning, "Correctable memory error rate exceeded for DIMM_B2". A correctable error is one that ECC memory (error-correcting code) caught and fixed on its own. One is normal. A rising rate of them is the first thing a failing module does.
That Warning row is easy to scroll past. It is not red. Nothing has stopped. But it is the whole reason you are here.
>>> Gather evidence now. Correctable errors that climb in rate are the runway before a multi-bit failure, the kind ECC cannot fix. The operator who catches the module here, while it is still only a Warning, replaces it on a schedule. The one who waits for Critical replaces it during an outage.
Keep watching. The correctable errors turn persistent, and then they cross the line ECC cannot hold. When that happens, the current-state plane finally catches up. Read the same DIMM.B2 health field you baselined at the start.
prompt: ops@dc-east-ws01:~$ answer: curl -s -u ADMIN:ADMIN http://$BMC_IP:5000/redfish/v1/Systems/1/Memory/DIMM.B2 | jq '.Status.Health' output: "Critical" hint: The exact same read as your baseline: curl -s -u ADMIN:ADMIN, the DIMM.B2 URL, a padded pipe, jq '.Status.Health'
The same field that read "OK" a few minutes ago now reads "Critical", live, without you touching the hardware. The system rollup at .Status.HealthRollup has gone Critical too, because Redfish propagates the worst health of any part up to the whole machine. The dashboard that said the node looks fine will now, at last, go red.
Read the recorder one more time and you have the complete history the module wrote as it died.
System Event Log
============================================================
ID Timestamp Sensor Event Severity
-------------------------------------------------------------------------------------------------------------------------------
1 07/30/2026 08:00:00 System_Power System boot completed Informational
2 07/30/2026 14:12:45 DIMM_B2 Correctable memory error rate exceeded for DIMM_B2 Warning
3 07/30/2026 14:13:30 DIMM_B2 Persistent correctable memory errors detected on DIMM_B2 Warning
4 07/30/2026 14:14:15 DIMM_B2 Multi-bit memory errors detected on a memory device at location DIMM_B2 Critical
Total entries: 4Four rows, in order: a clean boot, correctable errors, persistent correctable errors, then the multi-bit failure that flipped it Critical. That is the runway, start to finish.
Now put the two planes side by side. The diagram below is the same failure on both planes over time. Flip the toggle to see what you would have known if you had checked only once, at the very end.
{ "height": 460, "caption": "The same module failing on two planes over about three minutes. Flip to a single late check and watch the early window disappear.", "nodes": [ { "id": "n0r", "label": "Login: Redfish OK", "kind": "bmc", "x": 0, "y": 60, "detail": "At login the current-state plane reports DIMM.B2 healthy, Status Health OK. This is the baseline you record first." }, { "id": "n0s", "label": "Login: SEL quiet", "kind": "net", "x": 0, "y": 320, "detail": "At login the recorder holds one benign row, a clean boot at 08:00. Nothing has crossed a threshold yet." }, { "id": "n1s", "label": "+45s: correctable Warning", "kind": "net", "x": 300, "y": 320, "detail": "The first real signal. A correctable error rate Warning lands on the SEL. ECC is still fixing the errors, but the rate is climbing." }, { "id": "n2s", "label": "+90s: persistent Warning", "kind": "net", "x": 600, "y": 320, "detail": "The errors turn persistent. Same module, worse trend. Still only a Warning, still invisible to a single health check." }, { "id": "n3r", "label": "+135s: Redfish Critical", "kind": "bmc", "x": 900, "y": 60, "detail": "Now the current-state plane catches up. DIMM.B2 Health flips to Critical, and the system HealthRollup goes Critical with it." }, { "id": "n3s", "label": "+135s: multi-bit Critical", "kind": "net", "x": 900, "y": 320, "detail": "The recorder logs the multi-bit failure, the kind ECC cannot correct. This row and the Redfish flip happen together." }, { "id": "verdict", "label": "Replace DIMM.B2", "kind": "admin", "x": 1180, "y": 190, "detail": "Two independent planes name the same module. Safe to pull DIMM.B2 and file the warranty claim." } ], "edges": [ { "from": "n0r", "to": "n3r", "label": "health flips late", "kind": "mgmt" }, { "from": "n0s", "to": "n1s", "label": "first signal", "kind": "plain" }, { "from": "n1s", "to": "n2s", "label": "rate climbs", "kind": "plain" }, { "from": "n2s", "to": "n3s", "label": "crosses the line", "kind": "plain" }, { "from": "n3r", "to": "verdict", "label": "Redfish confirms", "kind": "mgmt" }, { "from": "n3s", "to": "verdict", "label": "SEL confirms", "kind": "oob" } ], "toggle": { "label": "When you looked:", "on": "Watched the whole window", "off": "Checked once, at the end", "dimOff": ["n0r", "n0s", "n1s", "n2s", "e:n0r-n3r", "e:n0s-n1s", "e:n1s-n2s", "e:n2s-n3s"] } }
You caught it. Not from a crash report written after the fact, but live, in the correctable-error window, the way the best operators do.
You now have every move this work order needs. Here is the cheat sheet you earned.
Work Order DC-EAST-WO-1042, six objectives. The BMC address is preset in $BMC_IP, Redfish on port 5000, credentials ADMIN / ADMIN.
1. Baseline the health. Save the DIMM.B2 reading while it is still OK. 2. Watch the log. Capture the running SEL so you can see it change. 3. The challenge. Catch the correctable-error Warning while the module is still only Warning. 4. Catch the flip. Capture the health the moment it reads Critical. 5. Confirm the failure. Save the SEL showing the multi-bit Critical row. 6. The challenge. File the RMA record with the failed module serial and part number.
Stuck is normal. Hit Request a signal on any objective for a nudge, or ask Daemon. It knows which task you are on and will nudge, not spoil.
Your workstation dc-east-ws01 is booting in the bay below, with the Bay 14 BMC on the management network beside it. This time the machine boots healthy. Within a few minutes, one module starts down the road you just traced. Launch it, watch it happen, and close Work Order DC-EAST-WO-1042.
Practice Live Fault Triage in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.