Learn › BMC & Server Management › Operations
A healthy-looking system rollup can still hide one failed DIMM. Diagnose a memory fault across both planes: read current component health over Redfish, confirm the failure history in the BMC SEL with getsel, and file the RMA evidence for DIMM.B2 on a live emulator.
It is 07:20 at the DC-EAST facility in Ashburn, Virginia. A monitoring alert just fired on the Dell PowerEdge R750xs in Bay 14, Row 7. The memory subsystem reads Critical. That is the whole alert.
Critical is a verdict, not an address. A DIMM (dual inline memory module) is one physical stick of RAM, and this server carries four of them. The alert does not say which one failed. Work Order DC-EAST-WO-1016 is on your tablet: find the failed module, confirm it on a second source, and file the evidence a warranty return needs.
Here is the trap you are about to walk past. A system health summary can read Critical while telling you almost nothing useful. It folds many parts into one word. Today you learn to distrust that one word and walk the tree until a single component owns the blame.
This is a Practice Zone. The diagrams below react to you: click the boxes, flip the switches. The real machine comes at the end, under Ready to practice.
A server this size answers to two management planes, and today you use both. Keep them straight, because they tell you different things.
Redfish is the modern management API (application programming interface). It is the BMC serving hardware facts as structured data over HTTP, and it reports the CURRENT state of every component: healthy, warning, or critical, right now.
The SEL (System Event Log) is the BMC black box recorder. It timestamps every threshold crossing as it happens and keeps the list on standby power. Where Redfish shows you the state now, the SEL shows you the story over time.
A careful engineer confirms a hardware fault on both planes before ordering a part. One plane can be misread. Two planes that agree are a diagnosis. Trace how they converge below.
{ "height": 420, "caption": "Two planes read the same module. Flip to Redfish only and watch the history disappear.", "nodes": [ { "id": "you", "label": "You (dc-east-ws01)", "kind": "admin", "x": 0, "y": 180, "detail": "Your workstation on the management network. Both planes answer here, over two different protocols." }, { "id": "rf", "label": "Redfish API :5000", "kind": "bmc", "x": 260, "y": 60, "detail": "The modern management API. It reports the current health of every component as structured JSON over HTTP." }, { "id": "sel", "label": "BMC SEL (getsel)", "kind": "net", "x": 260, "y": 300, "detail": "The System Event Log, the BMC black box. It timestamps every threshold crossing and keeps the history on standby power." }, { "id": "rollup", "label": "System health rollup", "kind": "host", "x": 560, "y": 60, "detail": "One word for the whole machine: Critical. It tells you a part failed, not which one." }, { "id": "b2", "label": "DIMM.B2 (failed)", "kind": "sensor", "x": 860, "y": 180, "detail": "The one failed module. Redfish localizes it by traversal; the SEL names it in the failure history. Both planes converge here." } ], "edges": [ { "from": "you", "to": "rf", "label": "GET + Basic auth", "kind": "mgmt" }, { "from": "you", "to": "sel", "label": "SSH getsel", "kind": "oob" }, { "from": "rf", "to": "rollup", "label": "current state", "kind": "plain" }, { "from": "rollup", "to": "b2", "label": "traverse to leaf", "kind": "plain" }, { "from": "sel", "to": "b2", "label": "history over time", "kind": "plain" } ], "toggle": { "label": "Planes in play:", "on": "Both planes", "off": "Redfish only", "dimOff": ["sel", "e:you-sel", "e:sel-b2"] } }
Flip the toggle to Redfish only. The event log path goes dark, and with it every clue about WHEN the module started to fail. Redfish alone can tell you a part is bad now; it cannot tell you the story that got it there.
>>> The SEL. Redfish is a snapshot of right now: this module is Critical. The SEL is the recording: correctable errors at 09:47, persistent errors overnight, multi-bit failure at 01:36. You need both, and you are about to read both.
Start on the Redfish plane. This BMC serves Redfish on port 5000, and every request carries HTTP Basic auth, the ADMIN / ADMIN credentials passed with the -u flag. The tool curl is the command-line HTTP client: it asks, the BMC answers in JSON (structured text with named fields).
Pull the whole system record and save it to a file, so you can read fields out of it without asking the BMC again. Type: curl -s -u ADMIN:ADMIN http://$BMC_IP:5000/redfish/v1/Systems/1 -o system.json
Now read the memory summary health out of that file. The tool jq is the JSON field scalpel: name a path, get the value. The memory verdict lives at .MemorySummary.Status.Health.
prompt: ops@dc-east-ws01:~$ answer: jq '.MemorySummary.Status.Health' system.json output: "Critical" hint: jq, then the path in single quotes: .MemorySummary.Status.Health, then the file system.json
There it is: the memory subsystem is Critical. The same verdict sits at the top level too, in .Status.HealthRollup, because Redfish propagates the worst health of any part up to the whole system. That propagation is useful and dangerous. It told you memory is bad. It did not tell you which of four DIMMs to pull.
The rollup pointed at memory. Now list the memory modules themselves. Redfish groups like components into a collection: a resource whose only job is to list its members and hand you the address of each one.
Pull the Memory collection and save it: curl -s -u ADMIN:ADMIN http://$BMC_IP:5000/redfish/v1/Systems/1/Memory -o memory.json
The collection holds a Members array, one entry per DIMM, each with an @odata.id link to that module page. List just those links. The -r flag prints raw text, without JSON quotes.
prompt: ops@dc-east-ws01:~$ answer: jq -r '.Members[]."@odata.id"' memory.json output: /redfish/v1/Systems/1/Memory/DIMM.A1 /redfish/v1/Systems/1/Memory/DIMM.A2 /redfish/v1/Systems/1/Memory/DIMM.B1 /redfish/v1/Systems/1/Memory/DIMM.B2 hint: jq -r, then the array path in single quotes: .Members[]."@odata.id", then memory.json
Four DIMMs: A1, A2, B1, B2. Read the collection closely and notice what is missing: not one health field. The collection lists the suspects, but it will not say which one is guilty. To get a per-module verdict you have to open each module own page, called a leaf.
A leaf is the endpoint page for one component: no children, just that part own facts. Open the leaf for DIMM.B2, the last module in the list, and save it: curl -s -u ADMIN:ADMIN http://$BMC_IP:5000/redfish/v1/Systems/1/Memory/DIMM.B2 -o dimm-b2.json
The leaf carries the module physical identity. Pull the fields that locate and name it.
prompt: ops@dc-east-ws01:~$ answer: jq '{Id, DeviceLocator, Manufacturer, MemoryDeviceType, CapacityMiB}' dimm-b2.json output: { "Id": "DIMM.B2", "DeviceLocator": "DIMM.Socket.B2", "Manufacturer": "Samsung", "MemoryDeviceType": "DDR5", "CapacityMiB": 32768 } hint: jq, then a filter in curly braces listing the five field names separated by commas, then dimm-b2.json
Read it like a technician holding a work order. DeviceLocator DIMM.Socket.B2 is the silkscreen label on the board, the exact socket to reach for. CapacityMiB 32768 is a 32 GB module. Read further down the same record and the leaf also reports Status.Health Critical, plus the SerialNumber and PartNumber a warranty return will ask for.
That is the Redfish plane verdict, localized at last: DIMM.B2 is the failed module. Open the leaf for a healthy neighbor like DIMM.A1 and its Status.Health reads OK. One bad module in four, and the rollup hid it behind a single word.
Redfish gave you the current state. Now cross to the other plane for the history. The SEL is not exposed the same way on this controller. The standard ipmitool sel list command does not surface these scenario events here, so you read the log through the BMC own shell instead. That difference is worth remembering: the two planes expose different things.
Open the Dell shell over SSH and run getsel. It prints the System Event Log in order, oldest entry first, straight off the controller black box.
prompt: /admin1-> answer: getsel output: System Event Log ============================================================
ID Timestamp Sensor Event Severity ------------------------------------------------------------------------------------------------------------------------------- 1 07/12/2026 03:11:04 System_Power System boot completed Informational 2 07/12/2026 09:47:19 DIMM_B2 Correctable memory error rate exceeded for DIMM_B2 Warning 3 07/12/2026 14:02:51 DIMM_B2 Correctable memory error rate exceeded for DIMM_B2 Warning 4 07/13/2026 01:36:08 DIMM_B2 Persistent correctable memory errors detected on DIMM_B2 Warning 5 07/13/2026 01:36:12 DIMM_B2 Multi-bit memory errors detected on a memory device at location DIMM_B2 Critical 6 07/13/2026 01:36:14 DIMM_B2 Memory device at location DIMM_B2 is critical Critical 7 07/13/2026 01:38:40 System_Power System boot completed Informational
Total entries: 7 hint: At the /admin1-> prompt, the one word that prints the System Event Log is getsel
Read the log top to bottom and you are reading a slow death, in order. DIMM_B2 first logs correctable errors, the kind that ECC memory (error-correcting code) quietly fixes. The rate climbs. The errors turn persistent. Then, at 01:36:12, they go multi-bit, past what ECC can correct, and the module is marked Critical two seconds later.
This is what the SEL gives you that Redfish cannot. Redfish said B2 is Critical now. The SEL says B2 started failing at 09:47 the day before and crossed the line overnight. Same module, named DIMM_B2 here with an underscore and DIMM.B2 in Redfish with a dot. Two planes, one culprit.
You now hold two independent readings that name the same part. That is the moment a diagnosis becomes safe to act on.
Why insist on two? Because acting on one is how good engineers pull healthy hardware. A misread field, a stale cache, a mislabeled socket: any single source can lie. When the Redfish leaf and the SEL history both point at DIMM.B2, the odds of both being wrong the same way collapse.
{ "height": 380, "caption": "One plane traverses to the module; the other confirms it. Flip off the second source and see the risk.", "nodes": [ { "id": "rollup", "label": "Rollup: Critical", "kind": "host", "x": 0, "y": 40, "detail": "The starting alert. Memory is Critical, but no module is named." }, { "id": "coll", "label": "Memory collection", "kind": "net", "x": 260, "y": 40, "detail": "Lists four DIMM members: A1, A2, B1, B2. Still no per-module health." }, { "id": "leaf", "label": "DIMM.B2 leaf", "kind": "sensor", "x": 520, "y": 40, "detail": "The page for this one module. Status Health Critical, plus its locator, serial, and part number." }, { "id": "selh", "label": "SEL history (getsel)", "kind": "bmc", "x": 260, "y": 260, "detail": "DIMM_B2 climbing from correctable errors to multi-bit Critical overnight. The independent second source." }, { "id": "verdict", "label": "Replace DIMM.B2", "kind": "admin", "x": 820, "y": 150, "detail": "Two independent readings name the same module. Now it is safe to pull the part and file the claim." } ], "edges": [ { "from": "rollup", "to": "coll", "label": "traverse", "kind": "plain" }, { "from": "coll", "to": "leaf", "label": "open leaf", "kind": "plain" }, { "from": "leaf", "to": "verdict", "label": "Redfish says B2", "kind": "mgmt" }, { "from": "selh", "to": "verdict", "label": "SEL agrees B2", "kind": "oob" } ], "toggle": { "label": "Second source:", "on": "SEL corroborates", "off": "Redfish alone", "dimOff": ["selh", "e:selh-verdict"] } }
The habit is the lesson. Rollup to collection to leaf on one plane, the event log on the other, and a written record at the end that cites both. That record is what a warranty desk accepts and what the next engineer trusts.
>>> Confirming on a second plane is cheap; pulling the wrong module is not. The SEL history and the Redfish leaf agreeing on DIMM.B2 is what turns a guess into a work order. That discipline is what this lab is really teaching.
Time to do it for real. Work Order DC-EAST-WO-1016: localize the memory fault on the Bay 14 R750xs and file the evidence. The BMC address is preset in $BMC_IP, Redfish on port 5000, credentials ADMIN / ADMIN.
Six objectives close this work order:
1. Read the rollup. Capture the memory subsystem health into ~/mem-health.json. 2. Count the suspects. Walk the Memory collection and see the four DIMMs. 3. Pin the module. Traverse to the failed leaf and save it to ~/dimm-b2.json. 4. Cross to the other plane. Read the SEL failure history into ~/sel-history.txt. 5. The challenge. Write a cross-plane diagnosis to ~/diagnosis.txt naming the module and the verdict. 6. The challenge. File the RMA part record to ~/rma-record.txt with the failed module serial number and part number.
Here is the deal: the workspace shows no commands. It hands you one objective at a time, and you recall the move. That recall is the whole point; it is how this hour reading becomes your skill.
Stuck is normal. Hit Request a signal on any objective for a nudge, or ask Daemon. It knows exactly which task you are on and will nudge, not spoil.
Your workstation dc-east-ws01 is booting in the bay below, with the Bay 14 BMC on the management network beside it and the memory alert still open. Launch it and close Work Order DC-EAST-WO-1016.
Practice Memory Fault Diagnosis in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.