Learn › BMC & Server Management › The Watchdog Timer
Final test: the whole Watchdog Timer chapter. Read a timer that was never set and keep the refusal a reload gets, name the five Timer Use values and the four timeout actions, arm one with a raw Set Watchdog Timer and decode its six data bytes, prove that arming alone does not start the countdown, start it and let it expire, capture the latched flag and the Watchdog2 record the controller wrote, let a power down action really take the host off, clear the latch, capture the pre-timeout this firmware refuses, and hand the machine back powered on with the timer stopped. 25 objectives on a real workstation with a real IPMI 2.0 controller at 10.100.0.2.
This chapter covers the following topics:
What you already know. Chapter 1 read mc watchdog get once on a machine that had never been powered and named the expiration flags as the difference between a watchdog reset and a crash. Chapter 2 established the difference between a reset, a power down and a power cycle. Chapter 3 read the event log and met the Pre-Init clock. Chapter 8 met three completion codes. This chapter arms the timer, starts it, lets it run out, and watches it power a host down.
Objectives. After this chapter you can:
The machine. This timer really acts. An expiry with the power down action opens node01's power circuit and chassis power status reads off afterwards; a power cycle really cycles it. Every output below was captured on the machine you will get. Boot takes about 3 minutes. This chapter is free; the machine is yours for 60 minutes and stops itself after 20 minutes idle.
This chapter uses a real Linux workstation and software-emulated IPMI hardware. The protocol requests and replies are real; controller identity, sensor values and host behavior may be seeded or emulated. Supplied physical-hardware captures are labeled separately.
These five questions establish whether you need this chapter in full. Each explanation names the section that covers the point.
>>> A watchdog is a countdown inside the controller that the host is responsible for reloading. The controller never reloads it, which is the whole point: a host that has stopped running cannot reload anything, and the controller acts on that silence. Section 9.1.
>>> The word reset in this command names the timer, not the machine. Reset Watchdog Timer reloads the countdown, which is what a watchdog daemon does every few seconds. chassis power reset is the command that restarts a host, and it is a different command in a different NetFn. Section 9.4.
>>> The expiration flags are a latch, not a live reading. They record that the timer ran out and which stage owned it, and they survive the action the controller took, which is what makes them evidence. Section 9.5.
>>> A configured timer is not necessarily running. Once a running countdown expires, the selected action is requested. A reboot loop depends on subsequent firmware, driver or daemon behavior; it is not guaranteed by one initial expiry. Section 9.9.
>>> Power down opens the power circuit. It is not a graceful request and nothing on the host is consulted, exactly as chassis power off behaved in Chapter 2. Power cycle is the separate action that takes the host off and back on. Section 9.6.
Goal: record the initial power state before testing a watchdog action. A watchdog is a controller-side countdown. Host software normally refreshes it; expiry triggers the configured action.
A refresh shows that the feeding path ran. It does not prove every application is healthy. The host-side driver or daemon must be configured to stop feeding when the condition it monitors fails.
prompt: ops@mgmt:~$ answer: ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 chassis power status output: Chassis Power is on hint: chassis power takes a status subcommand.
Chassis Power is on is the recorded starting state. Save the result for the assessment before any power-down or reset test.
Check: later, compare the final power state with this reading. Controller power state and operating-system health are different checks.
Now the timer itself, on a controller nobody has configured.
prompt: ops@mgmt:~$ answer: ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 mc watchdog get output: Watchdog Timer Use: Reserved (0x00) Watchdog Timer Is: Stopped Watchdog Timer Logging: On Watchdog Timer Action: No action (0x00) Pre-timeout interrupt: None Pre-timeout interval: 0 seconds Timer Expiration Flags: None (0x00) Initial Countdown: 0.0 sec Present Countdown: 4189.2 sec hint: mc watchdog has a get subcommand.
Nine lines, and this is a timer that has never been set.
Watchdog Timer Use : Reserved (0x00) is the giveaway. Section 9.2 covers the five real values; Reserved means none of them has been written.
Watchdog Timer Is : Stopped and Action : No action are the two fields that decide whether anything happens. Nothing would.
Timer Expiration Flags : None (0x00) is a latch, and it is empty because this timer has never run out.
Initial Countdown : 0.0 sec is what the timer would be loaded with.
Present Countdown : 4189.2 sec does not belong to any of that, and section 9.4.3 comes back to it. Your value will match, which is itself the finding.
Reloading a timer that was never configured is worth doing deliberately once, because the refusal is specific.
prompt: ops@mgmt:~$ answer: ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 mc watchdog reset output: Reset Watchdog Timer command failed: Attempt to reset uninitialized watchdog hint: mc watchdog has a reset subcommand, and it takes no arguments.
Attempt to reset uninitialized watchdog. The controller is refusing to start a countdown for a timer nobody has configured, which is correct: there is no action to take and no interval to count.
Objective 4 files this refusal, and like every refusal in this track it goes to standard error, so it needs 2>&1 or the file closes empty.
This refused reset does not initialize the timer. The later successful Set Watchdog Timer request supplies its configuration. Capture the uninitialized refusal before that step.
Chapter 8 used raw to establish where a refusal came from. The same command here gives the completion code by number.
prompt: ops@mgmt:~$ answer: ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 raw 0x06 0x22 output: Unable to send RAW command (channel=0x0 netfn=0x6 lun=0x0 cmd=0x22 rsp=0x80): Unknown (0x80) hint: raw takes the NetFn and the command number. Reset Watchdog Timer is NetFn 0x06, command 0x22, with no data bytes.
rsp=0x80, and it is a fourth completion code to add to the three Chapter 8 collected.
| Code | Meaning |
|---|---|
0x80 | Command-specific. Here: attempt to reset an uninitialized watchdog |
0xC1 | Invalid command in the request context |
0xCC | Invalid data field in the request; the field is not identified |
0xD4 | Insufficient privilege |
Codes 0x80 through 0xBE are command-specific: their meaning belongs to the command that returned them rather than being global. 0x80 from Reset Watchdog Timer means un-initialised; 0x80 from a different command means something else entirely. That is why the message matters as much as the number.
Note also the NetFn. The watchdog is an application command, NetFn 0x06, not a chassis one. It is not in the chassis device that Chapters 2 and 8 used, which is why mc info's capability list does not mention it.
Goal: read Timer Use as the configured purpose of the countdown. The expiration flag records that label; it does not by itself identify why feeding stopped.
prompt: ops@mgmt:~$ answer: printf '%s\n' 'Timer Use 1 is BIOS FRB2; 2 is BIOS/POST; 3 is OS Load; 4 is SMS/OS; 5 is OEM.' 'This lab configures SMS/OS. Firmware, drivers or daemons use the appropriate label as part of their platform policy.' 'An expiration flag identifies the configured use that expired. Correlate console, host logs and feeding policy before blaming firmware or the OS.' > ~/watchdog/timer-use.txt output: hint: Write the short evidence note with printf, then read it with cat ~/watchdog/timer-use.txt.
Your file should record these statements:
Check: identify the configured use and the observed action separately. An FRB2 flag is a diagnostic clue, not proof that every reset before the loader was caused by a POST fault.
The action is what the controller does when the countdown reaches zero. Chapter 2 met three of these four as power commands; here the controller performs one on its own.
prompt: ops@mgmt:~$ answer: printf 'The timeout action is what the controller does when the countdown reaches zero. Four values:\n0 No action. The expiry is latched and logged and the host is untouched.\n1 Hard reset. The controller drives the reset line: the machine restarts in place and power is never removed.\n2 Power down. The controller opens the power circuit and the host stops. Nothing is asked of the operating system.\n3 Power cycle. A power down followed by a power on.\nAll four are real on this controller and three of them move the host. Measured here: an expiry with power down really leaves chassis power status reading off.\n' > ~/watchdog/actions.txt output: hint: printf with one line per action and a closing sentence, separated by newline escapes, redirected into the file.
cat ~/watchdog/actions.txt:
The timeout action is what the controller does when the countdown reaches zero. Four values:
0 No action. The expiry is latched and logged and the host is untouched.
1 Hard reset. The controller drives the reset line: the machine restarts in place and power is never removed.
2 Power down. The controller opens the power circuit and the host stops. Nothing is asked of the operating system.
3 Power cycle. A power down followed by a power on.
All four are real on this controller and three of them move the host. Measured here: an expiry with power down really leaves chassis power status reading off.No action is not pointless. A watchdog set to log and do nothing is how a fleet is surveyed before anyone lets a timer restart machines: the expiries are recorded, nobody is woken, and the count says whether the timeout was set sensibly.
The other three map exactly onto Chapter 2. Hard reset is chassis power reset, power down is chassis power off, power cycle is chassis power cycle. The difference is who issues them: there, you did; here, the controller does, on its own, because a countdown reached zero.
Here is a gap worth noticing. ipmitool mc watchdog takes get, reset and off. There is no set. ipmitool can read a watchdog, reload one and stop one, and it cannot arm one.
So arming is a raw command: Set Watchdog Timer, NetFn 0x06, command 0x24, with six data bytes.
prompt: ops@mgmt:~$ answer: ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 raw 0x06 0x24 0x44 0x01 0x00 0x00 0x28 0x00 > ~/watchdog/arm.txt 2>&1 output: hint: raw with NetFn 0x06, command 0x24, then the six data bytes: timer use, action, pre-timeout interval, flags to clear, and the countdown low and high bytes. Redirect it, standard error included.
Run without the redirect, this command prints three lines of hex. They are not a result. Set Watchdog Timer returns no data bytes, and what this firmware prints is an uninitialised buffer: different bytes every run, sometimes a different number of lines, and meaningless either way. Do not read it, and do not put it in a report. Objective 7 files it because the objective is that you sent the command, not that the buffer said anything.
The six bytes you sent are 0x44 0x01 0x00 0x00 0x28 0x00, and section 9.3.2 decodes them. The command that says whether it worked is the read-back.
prompt: ops@mgmt:~$ answer: ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 mc watchdog get output: Watchdog Timer Use: SMS/OS (0x44) Watchdog Timer Is: Started/Running Watchdog Timer Logging: On Watchdog Timer Action: Hard Reset (0x01) Pre-timeout interrupt: None Pre-timeout interval: 0 seconds Timer Expiration Flags: None (0x00) Initial Countdown: 4.0 sec Present Countdown: 4189.2 sec hint: The same watchdog read as section 9.1.1.
Every field took. Timer Use is now SMS/OS (0x44) rather than Reserved, the Action is Hard Reset (0x01), and the Initial Countdown is 4.0 sec, which is the two countdown bytes decoded.
This is the read that Chapter 8's procedure called for and Chapter 6 met with an SNMP community string: a write that returned without an error is not a write until it has been read back.
Note what has NOT changed: Timer Expiration Flags is still None, and section 9.4 is about the other thing that has not changed.
Goal: explain the six bytes you sent, especially the running-state bit and the two different time units.
Work through the recorded value before writing it down:
0x44 is binary 01000100. Bit 6 is set, meaning do not stop an already running timer on this Set command. A stopped timer remains stopped. Bits 2:0 select SMS/OS; bit 7 is clear, so logging is not disabled.0x01 selects hard reset. The pre-timeout interrupt bits are zero.0x00 is the pre-timeout interval in whole seconds. The following 0x00 clears no expiration flags.0x28 0x00 is the countdown, low byte first: hexadecimal 0028 is 40 units of 100 milliseconds, or 4.0 seconds.Try it: record this decode using the guided command. It writes a text file; it does not change the watchdog configuration.
prompt: ops@mgmt:~$ answer: printf 'The six data bytes of Set Watchdog Timer, NetFn 0x06 command 0x24, and what I sent.\nbyte 1, timer use: 0x44. Bit 6 is SET: do not stop on set. An already stopped timer stays stopped; an already running timer continues with the new countdown. Bits 2:0 are 4, SMS/OS. Bit 7 would be do not log.\nbyte 2, action: 0x01, hard reset. Bits 6:4 carry the pre-timeout interrupt and are zero here.\nbyte 3, pre-timeout interval: 0x00, in WHOLE SECONDS.\nbyte 4, expiration flags to clear: 0x00, clear nothing. Bit 4 clears the SMS/OS flag.\nbytes 5 and 6, initial countdown: 0x28 0x00, low byte first, in HUNDRED-MILLISECOND units. 0x0028 is 40 units, which is 4.0 seconds, and that is what the read-back shows.\nTwo different units in one command is the trap: the pre-timeout is seconds and the countdown is tenths.\n' > ~/watchdog/set-bytes.txt output: hint: printf with one line per byte and the unit trap named, separated by newline escapes, redirected into the file.
Check: the file's first-byte explanation must say bit 6 is set in 0x44. Clearing that bit would instead tell Set Watchdog Timer to stop the timer automatically. Do not confuse changing configuration with starting a previously stopped timer.
The read-back's Initial Countdown : 4.0 sec agrees with 40 tenths of a second. The pre-timeout uses whole seconds, so those two fields must not be given the same unit conversion.
The assessment checks that the record explains timer use, countdown and its units. Writing an explanation is not proof that the controller started counting; the next exercises inspect that separately.
Goal: distinguish a configured timer from evidence that its countdown is running.
The previous read showed a four-second initial countdown. The Set request used 0x44: bit 6 was set, so it would not stop a timer that was already running. It also would not start one that was stopped. This lab began with a stopped, uninitialized timer.
Predict: if that timer remains stopped, waiting longer than four seconds should not produce a new expiry flag. The exercise waits twelve seconds and reads the timer; it does not send a reload.
prompt: ops@mgmt:~$ answer: sleep 12; ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 mc watchdog get output: Watchdog Timer Use: SMS/OS (0x44) Watchdog Timer Is: Started/Running Watchdog Timer Logging: On Watchdog Timer Action:
Practice The Watchdog Timer in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.