Learn › BMC & Server Management › Operations
The BMC's dead-man switch: learn how a hardware watchdog reboots a hung server all on its own, then inspect, configure, arm, evidence, and stand down the real timer from the controller shell on a live BMC.
It is 03:41 at the DC-EAST facility in Ashburn, Virginia. Two rows over from Bay 14, the kernel on a database node deadlocks mid-write. Every service on it freezes. No pager fires. No engineer logs in. And at 03:43, the node is booting clean, coming back to life on its own.
Nobody touched it. Nobody was even awake. Something inside the chassis noticed the operating system had gone silent, pulled the plug, and pushed it back in.
That something is the BMC's hardware watchdog timer, and tonight it is your work order. DC-EAST-WO-1009: the same protection must be armed on bay14-r750xs, the Dell PowerEdge R750xs in Bay 14, Row 7, primary PostgreSQL node of HA cluster DC-EAST-HA-07. The cluster carries a 99.99 percent SLA, which allows 52 minutes of downtime a year. A previous shift started the paperwork and never finished the job.
This is a Practice Zone. The diagrams below react to you: click the boxes, flip the switches. The real machine comes at the end, under Ready to practice.
Old locomotives had a pedal the driver held down while the train moved. Let go for too long and the brakes applied themselves, because a silent driver is an emergency, not a mystery. Engineers call that a dead-man switch: a device that acts on its own when a human, or a machine, stops proving it is alive.
The BMC watchdog is exactly that pedal. It is a countdown register inside the BMC: load it with a timeout, say 120 seconds, and it counts down toward zero. The running OS must pet the timer, meaning reset the countdown back to its full value, again and again before it runs out. A small daemon on the host does this every 30 seconds. As long as the petting continues, nothing ever happens.
If the kernel deadlocks, the daemon freezes with it. The petting stops. The countdown runs dry. And the BMC, which lives on its own processor and its own power, concludes the OS is dead and acts alone: it power-cycles the server.
{ "height": 380, "caption": "The loop that saved the 03:41 server. Click every hop, then flip the kernel.", "nodes": [ { "id": "you", "label": "You (NOC)", "kind": "admin", "x": 0, "y": 40, "detail": "You configure the switch once, from your desk. After that, recovery needs no human anywhere on the path. That is the point of a dead-man switch." }, { "id": "bmc", "label": "BMC", "kind": "bmc", "x": 330, "y": 40, "detail": "The always-on controller you know. The watchdog countdown lives inside it, on separate power, out of the kernel's reach." }, { "id": "timer", "label": "Watchdog timer 120s", "kind": "sensor", "x": 620, "y": 40, "detail": "A countdown register inside the BMC. Every pet refills it to 120 seconds. If it ever reaches zero, the BMC acts alone." }, { "id": "power", "label": "Power control", "kind": "psu", "x": 620, "y": 300, "detail": "The BMC's hand on the server's power. It works even when the CPU is locked solid." }, { "id": "host", "label": "Host OS", "kind": "host", "x": 60, "y": 300, "detail": "Runs PostgreSQL, and the watchdog daemon along with it. If the kernel hangs, the daemon hangs too, and the petting stops." }, { "id": "daemon", "label": "Watchdog daemon", "kind": "host", "x": 330, "y": 300, "detail": "A small OS service that opens /dev/watchdog and pets the timer every 30 seconds. Each pet refills the countdown to 120." } ], "edges": [ { "from": "you", "to": "bmc", "label": "arm / stand down (shell)", "kind": "oob" }, { "from": "bmc", "to": "timer", "label": "holds the countdown", "kind": "plain" }, { "from": "host", "to": "daemon", "label": "runs", "kind": "plain" }, { "from": "daemon", "to": "timer", "label": "pet every 30 s", "kind": "mgmt" }, { "from": "timer", "to": "power", "label": "expiry: power-cycle", "kind": "power" }, { "from": "power", "to": "host", "label": "hard reboot", "kind": "power" } ], "toggle": { "label": "Host kernel:", "on": "Healthy (petting)", "off": "Hung at 03:41 (silent)", "dimOn": ["e:timer-power", "e:power-host"], "dimOff": ["e:daemon-timer"] } }
Flip the kernel switch and watch the loop change hands. Healthy kernel: the pet edge is alive and the kill path stays dark. Hung kernel: the pet goes silent and the expiry path lights up, timer to power to a hard reboot. No human appears anywhere on the recovery side.
>>> Petting stopped at 03:41. The countdown ran dry at 03:43. The BMC acted with no human anywhere on the path. That autonomy is the whole point of a dead-man switch, and, as you will see at the end of this lesson, the whole risk.
A fair question: why involve the BMC at all? Linux already has a software watchdog. Load the softdog kernel module and the kernel will watch for hangs all by itself.
Look at where each guard lives before you trust it. A software watchdog is a kernel module: it lives inside the very thing it is supposed to guard. If an application crashes while the kernel stays healthy, softdog can catch it and restart services. But when the kernel itself panics, everything the kernel hosts dies in the same instant, including its watchdog.
The BMC watchdog lives on the other side of the wall: separate processor, separate firmware, standby power. A kernel panic cannot touch it. For exactly the failure that matters most, it is the only guard left standing.
{ "height": 320, "caption": "Two guards, one machine. Flip the failure and see which guard survives.", "nodes": [ { "id": "kernel", "label": "Kernel", "kind": "host", "x": 60, "y": 40, "detail": "The core of the OS. When it panics, everything it hosts stops with it, including any software that was supposed to help." }, { "id": "softdog", "label": "softdog (software)", "kind": "host", "x": 330, "y": 40, "detail": "A software watchdog is a kernel module. It lives inside the very thing it is supposed to guard." }, { "id": "server", "label": "Server", "kind": "host", "x": 700, "y": 170, "detail": "The machine both watchdogs claim to protect." }, { "id": "bmc", "label": "BMC watchdog", "kind": "bmc", "x": 60, "y": 280, "detail": "Separate processor, separate firmware, standby power. A kernel panic cannot touch it." }, { "id": "power", "label": "Power control", "kind": "psu", "x": 380, "y": 280, "detail": "The BMC's power switch works no matter what state the kernel is in." } ], "edges": [ { "from": "kernel", "to": "softdog", "label": "hosts", "kind": "plain" }, { "from": "softdog", "to": "server", "label": "can restart services", "kind": "inband" }, { "from": "bmc", "to": "power", "label": "hard power-cycle", "kind": "power" }, { "from": "power", "to": "server", "kind": "power" } ], "toggle": { "label": "Failure:", "on": "App crash (kernel alive)", "off": "Kernel panic", "dimOn": [], "dimOff": ["softdog", "e:kernel-softdog", "e:softdog-server"] } }
Flip the failure to a kernel panic and watch the software watchdog dim: it died with its host. Only the out-of-band path survives.
>>> A software watchdog lives inside the thing it guards, so a kernel panic kills the guard along with the prisoner. The BMC watchdog survives because it is out-of-band hardware. That is why the IronCore HA policy names the BMC watchdog, not softdog, as mandatory equipment on every cluster node.
Time to look at the actual switch on bay14-r750xs. Two views exist, and a careful operator reads both. The first is the live register the BMC exposes over the network. The watchdog lives under mc in ipmitool, the management controller family you already know, and you read it the same way you read everything else on this BMC. $BMC_IP is preset in the lab.
prompt: ops@dc-east-ws01:~$ answer: ipmitool -I lanplus -C 3 -H $BMC_IP -U ADMIN -P ADMIN mc watchdog get output: Watchdog Timer Use: Reserved (0x00) Watchdog Timer Is: Stopped Watchdog Timer Logging: On Watchdog Timer Action: No action (0x00) Pre-timeout interrupt: None Pre-timeout interval: 0 seconds Timer Expiration Flags: None (0x00) Initial Countdown: 0.0 sec Present Countdown: 4189.2 sec hint: Type it exactly: ipmitool -I lanplus -C 3 -H $BMC_IP -U ADMIN -P ADMIN mc watchdog get
Decode it line by line:
One honest quirk of this particular BMC: the two live-state lines never update. Watchdog Timer Is: always reads Stopped, and whatever number Present Countdown: shows is stale junk that never ticks. This read is the network-side snapshot, and it is read-only from here: on this bench the register cannot be written over IPMI. The switch itself is armed from the controller's own shell, which is where you go next.
The second view lives inside the BMC. The controller keeps its own stored watchdog record, and, crucially on this bench, that record is the one you actually configure. On a production iDRAC you compare both views before touching anything. Get the habit now.
SSH into the controller the way you did in First Contact: sshpass -p ADMIN ssh -o StrictHostKeyChecking=no admin@$BMC_IP. At the /admin1-> prompt, ask for the timer by name.
prompt: /admin1-> answer: watchdog output: Watchdog Timer Configuration ============================================================
Timer Use: SMS/OS Timer State: Stopped Timer Action: No action Pre-timeout Interrupt: None Timeout (seconds): 0 Current Countdown: 0 hint: One word: watchdog
Read the story in that table. Timer Use: SMS/OS means someone already chose the right owner for this timer. SMS stands for System Management Software, so SMS/OS marks the timer as belonging to the running operating system and its watchdog daemon. That choice was the previous shift's work.
Now read the rest: no action, no timeout, stopped. The paperwork exists; the protection does not. And unlike the network register, this stored record is writable from right here: the same watchdog command that displays it also configures it. Your job tonight is to finish what the last shift started, from inside this shell. Type exit when you want to leave the controller.
Everything you do to this watchdog is one command, watchdog, with three moves, all run at the /admin1-> prompt inside the controller. The HA spec at /etc/lab/docs/ha-cluster-spec.txt on your workstation dictates every value you load.
Move one: load the spec. watchdog set takes a timeout in seconds and an expiry action:
120: the spec requires auto-recovery within 120 seconds, and the timeout is given in seconds.cycle: a full power-cycle clears wedged hardware where a plain reset might not, so the spec demands cycle. The timer's use is already SMS/OS, the OS-side owner, from the previous shift.prompt: /admin1-> answer: watchdog set 120 cycle output: Watchdog Timer configured successfully. Timer Action: Power Cycle Timeout: 120 seconds Timer State: Stopped (loaded, not armed) hint: At the /admin1-> prompt: watchdog set 120 cycle
Configured is not armed. Per the IPMI standard, a set loads the countdown but does not start it. A loaded timer sits quiet until something starts it, which is why the state still reads Stopped.
Move two: arm it. watchdog arm starts the countdown. In production the OS watchdog daemon effectively does this over and over, refilling the countdown before it can ever reach zero.
prompt: /admin1-> answer: watchdog arm output: Watchdog Timer armed - countdown started. Timer State: Running Countdown: 120 seconds hint: At the /admin1-> prompt: watchdog arm
The stored record now reads Timer State: Running. That word is the proof the switch is live: when you file evidence later, Running is what the cluster change record must show.
Move three: stand down. For maintenance, watchdog off stops the countdown and drops the action back to No action. Why would you ever disarm your own protection? Flip the maintenance window below.
{ "height": 340, "caption": "The BMC cannot tell a flash from a hang. Flip the maintenance window.", "nodes": [ { "id": "you", "label": "NOC (you)", "kind": "admin", "x": 0, "y": 160, "detail": "The person holding the dead-man switch. Knowing when to take its finger off the trigger is part of the job." }, { "id": "bmc", "label": "BMC watchdog", "kind": "bmc", "x": 280, "y": 160, "detail": "It cannot tell a firmware flash from a kernel hang. Silence is silence." }, { "id": "power", "label": "Power control", "kind": "psu", "x": 540, "y": 160, "detail": "The expiry path. Fatal to a machine caught mid-flash." }, { "id": "host", "label": "bay14-r750xs", "kind": "host", "x": 800, "y": 160, "detail": "Primary PostgreSQL node of DC-EAST-HA-07. 99.99 percent SLA: 52 minutes of downtime a year." }, { "id": "flash", "label": "Firmware flash", "kind": "net", "x": 540, "y": 20, "detail": "BIOS and iDRAC updates pause the OS for minutes. The petting stops even though nothing is wrong." } ], "edges": [ { "from": "you", "to": "bmc", "label": "watchdog off / arm", "kind": "oob" }, { "from": "bmc", "to": "power", "label": "expiry: power-cycle", "kind": "power" }, { "from": "power", "to": "host", "kind": "power" }, { "from": "flash", "to": "host", "label": "OS paused, no petting", "kind": "plain" } ], "toggle": { "label": "Maintenance window:", "on": "Closed (watchdog armed)", "off": "Open (watchdog disarmed)", "dimOn": ["flash", "e:flash-host"], "dimOff": ["e:bmc-power", "e:power-host"] } }
With the window open and the watchdog still armed, the flash pauses the OS, the petting stops, and the expiry path fires straight into a machine that is halfway through rewriting its own firmware. Disarming dims the kill path for exactly as long as the work takes.
>>> The BMC cannot tell a firmware flash from a kernel hang. Silence is silence, and a power-cycle in the middle of a flash can brick the board. Disarm, flash, re-arm is standard change procedure on every serious floor, and it is exactly your final task on the real machine.
A word of respect for the real thing. On production hardware, arming a watchdog is done with care: you confirm the OS petting daemon is running first, you set a timeout long enough that heavy I/O cannot trip a false reboot, and you disarm before every maintenance window. An armed watchdog with no daemon petting it will reboot a perfectly healthy machine on schedule. Here on the bench you are the daemon, and the cost of a mistake is only a re-arm; in the rack, the cost is a production reboot. Same commands, higher stakes.
One sentence for a later module: real iDRACs also expose this timer over Redfish as HostWatchdogTimer, but on this bench the watchdog is configured through the controller shell.
How this bench maps to a real iDRAC. The stored watchdog record you configure here is the real timer's configuration table: the same Timer Use, action, timeout, and Running state you would set on production hardware, with the same load-then-arm state machine. Three honest gaps: the network-side register (ipmitool mc watchdog get) is read-only on this bench and does not mirror the shell changes, the shell record shows Running but does not animate a ticking countdown, and when the timer fires it power-cycles an emulated chassis, not a live host. The commands, the sequence, and the muscle memory are identical.
Time to do it for real. Work Order DC-EAST-WO-1009 closes when the dead-man switch on bay14-r750xs has been inspected, armed to spec, evidenced, and stood down for tonight's maintenance window. $BMC_IP is preset, credentials ADMIN / ADMIN, and the HA spec waits at /etc/lab/docs/ha-cluster-spec.txt.
Six objectives:
1. Read the live register over the network and confirm it is factory-blank. 2. Pull the stored record from inside the controller's own shell. 3. Load the HA spec into the timer from the shell: the right action, the right countdown. 4. Arm it from the shell. From that moment, the switch is live. 5. File the armed readout to ~/watchdog-armed.txt for the cluster change record. 6. CHALLENGE: the firmware maintenance window opens. Stand the switch down and file the post-standdown readout to ~/watchdog-standdown.txt. No hints on this one.
Here is the deal: the workspace shows no commands. It hands you one objective at a time, and you recall the move. That recall is the whole point; it is how tonight's reading becomes your skill.
Stuck is normal. Hit Request a signal on any objective for a nudge, or ask Daemon. It knows exactly which task you are on and will nudge, not spoil.
Your workstation dc-east-ws01 is booting in the bay below, with the Dell BMC on the management network beside it. Somewhere two rows over, a database node owes its uptime to the switch you are about to arm. Launch, and close the work order.
Practice Watchdog Timer in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.