Learn › BMC & Server Management › The System Event Log

The System Event Log

Final test: the whole System Event Log chapter. Read the System Event Log with ipmitool, sel info, sel list and sel elist, slice an eight-event overnight incident by subsystem, name an uncorrectable-memory-ECC root cause and separate it from a survivable power-supply red herring, read the Pre-Init clock and why a real BMC syncs it with NTP, dump a record in full, file the genuine Dell iDRAC reference and the vendor dialects, and record the verdict without clearing the evidence. 25 objectives on a real workstation called mgmt with a real IPMI 2.0 controller at 10.100.0.2 holding last night's incident.

Chapter 3. The System Event Log

This chapter covers the following topics:

3.1 The system event log and what writes it
3.2 Record forms: sel list and sel elist
3.3 Filtering and counting a log by subsystem
3.4 Reading one record in full
3.5 The controller clock and why record order is the timeline
3.6 Building a timeline from a log
3.7 Root cause and the events that merely accompanied it
3.8 Severity, and the verdict a log supports
3.9 When a log may be cleared
3.10 Production logs and vendor implementations

What you already know. Chapter 1 read sel info, sel list and sel elist on a controller holding a single thermal excursion, and established that assertions and deassertions pair. Chapter 2 established that this controller does not log power actions of its own. This chapter reads a log written by a real incident and reaches a conclusion from it.

Objectives. After this chapter you can:

Read an event log summary and say what its capacity and overflow behaviour mean for evidence
Select records by subsystem and count them without reading the whole log
Read a single record in full and name what each field carries
Establish a timeline from record order when the controller clock is not set
Separate a root cause from the events that accompanied it
State whether hardware is safe to keep in service, and support the statement from the log
Say when an event log may be cleared and when it may not

The machine. The workstation is a real Linode running Ubuntu 24.04, and the controller beside it is a software IPMI 2.0 controller. The eight records read in this chapter were written through the controller's own event engine, so every read here is a real system event log read. Boot takes about 3 minutes. This chapter is free; the machine is yours for 60 minutes and stops itself after 20 minutes idle.

This chapter uses a real Linux workstation and software-emulated IPMI hardware. The protocol requests and replies are real; controller identity, sensor values and host behavior may be seeded or emulated. Supplied physical-hardware captures are labeled separately.

Pre-check 1 of 5

These five questions establish whether you need this chapter in full. Each explanation names the section that covers the point.

[ ] Nothing; the log discards its oldest records automatically
[x] The log is nearly full, and depending on firmware the next events are either dropped or overwrite the oldest ones
[ ] The controller will refuse new sessions until the log is cleared
[ ] The overflow flag is wrong, since a log at 98% has already overflowed

>>> The log is a fixed-size store. Some firmware stops accepting records when it is full and some overwrites the oldest; either way, a log left near capacity loses the evidence of the next incident. Overflow : false says it has not happened yet. Section 3.1.

Pre-check 2 of 5

[ ] The power supply AC loss, because it is the only external event
[x] The uncorrectable memory error, with the correctable one as the warning that preceded it
[ ] The correctable memory error, because it happened first
[ ] It cannot be determined without wall-clock timestamps

>>> A correctable error is repaired in hardware and the machine continues. An uncorrectable one cannot be repaired and takes the host down. The AC loss on one supply of a redundant pair is survivable and the machine survived it. The correctable error is the same part failing slowly. Section 3.7.

Pre-check 3 of 5

[ ] Nothing; the log is unusable as evidence
[x] The retained log order, by following the returned entries rather than sorting their IDs numerically
[ ] The elapsed time between events, by subtracting the record identifiers
[ ] Only the total count

>>> Pre-Init means the controller had no clock set when the records were written. The retained entries form an ordered list, but record IDs are handles and need not be numerically ordered. Preserve the reader's traversal order; it does not give elapsed time or establish causation. Section 3.5.

Pre-check 4 of 5

[ ] Immediately on arriving at an incident, so that new events are easy to see
[ ] Never; the command exists only for firmware development
[x] After the records have been captured and the incident is understood, so the next reading starts clean
[ ] Whenever Percent Used passes 50%, as routine maintenance

>>> sel clear is irreversible and the log is the evidence. It is run after the records are saved and the fault is resolved, which leaves a clean log for the next incident without destroying the current one. Section 3.9.

Pre-check 5 of 5

[ ] The named record is newer, and older records lose their names
[x] elist resolves a name only where the sensor data repository holds a matching record
[ ] The unnamed record was written by the host rather than by the controller
[ ] The controller is truncating output for records over a certain length

>>> elist looks each sensor number up in the sensor data repository. Where a record exists it prints the name and the reading; where none exists it prints the number, which still identifies the subsystem. Section 3.2.

3.1 The system event log

The system event log, SEL, is a fixed-size store inside the controller. The controller writes it, not the operating system, which is why it survives a crash, a reinstall and a machine that never finished booting. On a host that restarted with nobody watching, it is the only account of what happened.

sel info summarises it before you read a single record.

prompt: ops@mgmt:~$ answer: ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sel info output: SEL Information Version : 1.5 (v1.5, v2 compliant) Entries : 8 Free Space : 15872 bytes Percent Used : 0% Last Add Time : S+ 00:00:46 Last Del Time : Not Available Overflow : false Supported Cmds : 'Delete' 'Reserve' hint: sel has an info subcommand.

Entries : 8 is the record count, and it is the number to check against your own count later.

Free Space and Percent Used describe a store with a hard limit. What happens at the limit depends on firmware: some controllers refuse new records, some overwrite the oldest. Either behaviour loses evidence, and a log left near capacity is a log that will not hold the next incident.

Overflow : false says the limit has not been reached. true means records have already been lost, and everything read below it is incomplete.

Supported Cmds : 'Delete' 'Reserve' says the log accepts a clear. A log without Delete cannot be cleared except through a firmware operation.

Last Add Time is when the most recent record arrived. Your value will differ; section 3.5 explains why it reads as an offset rather than a date.

3.2 The records

sel list prints the records in the compact form: identifier, timestamp, a hexadecimal field, the sensor, the event, and the direction.

Eight records, and read as a set they already tell most of the story. Read them in identifier order:

1. A correctable ECC error on a memory device. Correctable means the error-correcting code repaired it and the machine continued. It is a warning, not a failure. 2. Power Supply AC lost on supply 2. One feed of a redundant pair stopped delivering. 3. Predictive failure on the same supply. 4. A temperature sensor crossing its upper critical threshold. 5. An uncorrectable ECC error on the same memory device as record 1. Uncorrectable means the code could not repair it. 6. A critical interrupt, an NMI, which is what a machine raises as it fails. 7. Power Unit, Power off/down: the machine stopped. 8. System Boot Initiated by power up: it started again.

Note what is not here: nothing says "root cause". That is the reading you produce, and sections 3.6 and 3.7 produce it.

3.2.1 The extended form

sel elist reads the same records and resolves each sensor number against the sensor data repository.

One record changed. Record 4 has become Temperature Exhaust_Temp and gained Reading 51 > Threshold 45 degrees C: the sensor name, the value it reached, and the limit it crossed.

The other seven are unchanged, still identified by number. That is not a failure of the command, and section 3.2.2 explains it.

sel elist is the form to reach for, because where it can resolve a record it tells you materially more, and where it cannot it tells you exactly what sel list would have.

3.2.2 Why some records resolve and some do not

sel list shows log records with sensor numbers. sel elist also consults the sensor data repository (SDR) to interpret them.

When a matching SDR exists, the extended output can show the sensor's name and decoded reading. Without a matching SDR, a number such as Memory #0x0a can remain. The number and event text are still evidence; a missing friendly name does not mean the event did not happen.

This controller carries sensor records for five sensors, so only those five resolve. Objective 22 records the explanation:

prompt: ops@mgmt:~$ answer: printf 'sel elist resolves a sensor NAME only when a matching SDR exists in the repository. For the five sensors with SDRs, elist shows the name; for the others it shows the sensor number, which is honest. That is why some records read as a number and some as a name.\n' > ~/sel/names.txt output: hint: printf with the explanation in single quotes and a trailing newline escape, redirected into the file.

The file reads:

sel elist resolves a sensor NAME only when a matching SDR exists in the repository. For the five sensors with SDRs, elist shows the name; for the others it shows the sensor number, which is honest. That is why some records read as a number and some as a name.

A bare number is still evidence. Memory #0x0a names the subsystem and the device, and the event text still says what happened to it. What is missing is the friendly name and the reading, neither of which changes the conclusion here.

3.3 Counting the log

sel info reported eight entries. Confirming that from the records themselves is one pipe. grep -cF '|' counts lines containing a literal vertical bar, which every record line has, and the brace group puts a label in front of the number:

prompt: ops@mgmt:~$ answer: { printf 'total events in the log: '; ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sel list | grep -cF '|'; } > ~/sel/count.txt output: hint: Group a printf label and the sel list piped into grep counting a literal bar, then redirect the group.

The file reads:

total events in the log: 8

-c makes grep print a count instead of the matching lines. -F makes it treat the pattern as a fixed string rather than a regular expression, which states our intent explicitly. In default basic grep, an unescaped | is already literal; with grep -E it becomes an alternation operator.

Eight, matching the summary. A count from the records that disagrees with the summary means records were added or removed while you were reading, which is worth knowing.

3.3.1 Filtering by subsystem

A production log holds hundreds of records and the useful question is always about one subsystem. grep -i matches case-insensitively:

prompt: ops@mgmt:~$ answer: ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sel list | grep -i 'ecc' > ~/sel/memory.txt output: hint: Pipe the log into a case-insensitive grep for ecc and redirect.

The file reads:

   1 |  Pre-Init  |0000000045| Memory #0x0a | Correctable ECC | Asserted
   5 |  Pre-Init  |0000000046| Memory #0x0a | Uncorrectable ECC | Asserted

Two records, both on Memory #0x0a: the same device, correctable first and uncorrectable later. That pairing on one device is the strongest single signal in this log, and it took one command to isolate.

ECC, error-correcting code, is the memory technology that detects and repairs errors. A correctable error is one the code repaired; the machine continues and the event is a warning about the device. An uncorrectable error is one the code detected and could not repair, and the only safe response is to stop.

3.3.2 Counting one subsystem

The same pattern with -c gives a count rather than the records, which is what a report wants:

prompt: ops@mgmt:~$ answer: { printf 'memory ECC events in the log: '; ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sel list | grep -ci 'ecc'; } > ~/sel/mem-count.txt output: hint: The same brace group as the total count, with grep counting case-insensitively for ecc.

The file reads:

memory ECC events in the log: 2

-ci combines the count and the case-insensitive match. Two on this log; on a machine with a genuinely failing device the count climbs over days, and the rate of climb is what decides how urgent the replacement is.

3.3.3 The power supply records

prompt: ops@mgmt:~$ answer: ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sel list | grep -i 'power supply' > ~/sel/psu.txt output: hint: Pipe the log into a case-insensitive grep for the power supply text and redirect.

The file reads:

   2 |  Pre-Init  |0000000045| Power Supply #0x02 | Power Supply AC lost | Asserted
   3 |  Pre-Init  |0000000045| Power Supply #0x02 | Predictive failure | Asserted

Two records on supply 2. Power Supply AC lost means that supply stopped receiving mains; the usual causes are a pulled cord, a failed feed, or a supply that has died. Predictive failure is the supply reporting that it expects to fail.

These are real and they need a response. They are also not why the machine went down: a redundant pair is designed so that one supply carries the load alone, and the machine kept running through both records. Section 3.7 returns to this.

3.3.4 The thermal record

prompt: ops@mgmt:~$ answer: ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sel list | grep -i 'temperature' > ~/sel/thermal.txt output: hint: Pipe the log into a case-insensitive grep for temperature and redirect.

The file reads:

   4 |  Pre-Init  |0000000046| Temperature #0x03 | Upper Critical going high | Asserted

One record, an upper critical crossing on Temperature #0x03. The extended form in section 3.2.1 named it Exhaust_Temp and gave the numbers: a reading of 51 against a limit of 45.

It has no matching deassertion, which means the temperature had not come back down by the time the machine stopped. Chapter 1 established that assertions pair with deassertions when a condition clears; an assertion with no partner is a condition that was still true.

3.3.5 The restart itself

Two records describe the machine stopping and starting. A single grep with alternation catches both. -E enables extended regular expressions, in which | means or:

prompt: ops@mgmt:~$ answer: ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sel list | grep -iE 'boot|power off|power down|power unit' > ~/sel/reboot.txt output: hint: Pipe the log into grep with -iE and the four alternatives separated by vertical bars, then redirect.

The file reads:

   7 |  Pre-Init  |0000000046| Power Unit #0x01 | Power off/down | Asserted
   8 |  Pre-Init  |0000000046| System Boot Initiated #0x01 | Initiated by power up | Asserted

Record 7 is the machine stopping. Record 8 is System Boot Initiated | Initiated by power up, and that phrase is the finding: the machine came back by a power-up, not by a reset and not by an operator. It restarted itself.

Note that -E and -F are opposites. -F in section 3.3 made the bar a literal character; -E here makes it an operator. Choosing the wrong one is the usual reason a grep against a log returns everything or nothing.

3.3.6 The earliest record

The first record in the log is where a timeline starts. head -1 keeps the first line:

prompt: ops@mgmt:~$ answer: ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sel list | head -1 > ~/sel/first.txt output: hint: Pipe the log into head keeping one line, then redirect.

The file reads:

   1 |  Pre-Init  |0000000045| Memory #0x0a | Correctable ECC | Asserted

The earliest record is the correctable memory error. On an incident this compact the first record is often the first symptom, and here it is: the device that eventually failed had already reported one repaired error.

On a log that has wrapped, the earliest record is not the earliest event, which is what Overflow in section 3.1 warns about.

3.4 One record in full

sel get takes a record identifier and prints every field of that record. 2>&1 sends standard error into the same file, so a refusal is captured rather than lost:

prompt: ops@mgmt:~$ answer: ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sel get 1 > ~/sel/record1.txt 2

Practice The System Event Log in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.