Learn › BMC & Server Management › Operations
No SSH, no monitor, no crash cart: reach a dead machine's serial console over the management network with Serial-over-LAN, then capture the boot failure that explains everything.
The phone buzzes at 02:47. The NOC board shows bay14-r750xs, the Dell PowerEdge R750xs in Bay 14, Row 7, dark for 23 minutes. The BMC reports POST completed and handed off to the boot loader. Then nothing. The OS never came up, so there is no IP address, no SSH, no monitoring agent. The machine is powered on, working hard at failing, and completely unreachable.
The old answer is a crash cart: a monitor and keyboard on wheels that someone pushes across the data hall, through two badge doors, to plug into the server's VGA port. Forty minutes, minimum, and it assumes someone is even in the building at 02:47.
Work Order DC-EAST-WO-1008 asks for the new answer: see that machine's screen without leaving your desk. The feature that makes it possible is called Serial-over-LAN, and it lives inside the BMC you already know.
This is a Practice Zone. The diagrams below react to you: click the boxes, flip the switches. The real machine comes at the end, under Ready to practice.
Every server still ships with a serial port. Not out of nostalgia: for exactly this night. A serial port (the chip behind it is called a UART, and on this Dell the port is ttyS1, also known as COM2) moves plain text one character at a time. That simplicity is the superpower. Text over serial works before video drivers load, before the OS starts, before networking exists. The BIOS can write to it. GRUB can write to it. A crashing kernel can write to it.
SSH cannot make that claim. SSH needs a booted OS with a working network stack, which is precisely what bay14-r750xs does not have tonight. Trace both doors below, then flip the host state and watch which one survives.
{ "height": 360, "caption": "Flip the host state. One door dies with the OS; the other never closes.", "nodes": [ { "id": "you", "label": "You (NOC)", "kind": "admin", "x": 0, "y": 170, "detail": "Your workstation at the NOC. Two possible roads to the machine in Bay 14." }, { "id": "msw", "label": "Management sw", "kind": "net", "x": 240, "y": 40, "detail": "The management switch for Row 7. Alive no matter what the host is doing." }, { "id": "bmc", "label": "BMC", "kind": "bmc", "x": 480, "y": 40, "detail": "The always-on controller. It owns UDP port 623 and wraps the serial stream into network packets." }, { "id": "uart", "label": "ttyS1 (COM2)", "kind": "nic", "x": 700, "y": 170, "detail": "The host serial port. Firmware and kernel write boot text here long before networking exists." }, { "id": "host", "label": "bay14-r750xs", "kind": "host", "x": 920, "y": 170, "detail": "POSTs fine, hangs in GRUB, never brings up an IP. The in-band door is dead on arrival." }, { "id": "psw", "label": "Production sw", "kind": "net", "x": 240, "y": 300, "detail": "The production switch. Perfectly healthy, and completely useless tonight." }, { "id": "eth0", "label": "eth0", "kind": "nic", "x": 480, "y": 300, "detail": "The host network port. Dead until the OS boots the network stack, which is exactly what never happens." } ], "edges": [ { "from": "you", "to": "msw", "label": "SOL session", "kind": "mgmt" }, { "from": "msw", "to": "bmc", "kind": "mgmt" }, { "from": "bmc", "to": "uart", "label": "serial over LAN", "kind": "oob" }, { "from": "uart", "to": "host", "label": "console text", "kind": "plain" }, { "from": "you", "to": "psw", "label": "SSH (dead)", "kind": "inband" }, { "from": "psw", "to": "eth0", "kind": "inband" }, { "from": "eth0", "to": "host", "kind": "inband" } ], "toggle": { "label": "Host state:", "on": "OS booted", "off": "Hung in GRUB", "dimOff": ["psw", "eth0", "e:you-psw", "e:psw-eth0", "e:eth0-host"] } }
When the host hangs in GRUB, the whole bottom lane goes dark: no OS, no network stack, no eth0, no SSH. The top lane does not care. The BMC is awake on standby power, the UART is wired straight into the host board, and boot text is flowing whether anyone reads it or not. Serial-over-LAN (SOL) is the BMC reading that text and shipping it to your terminal over the management network.
SOL is a pipeline with two halves, and both must be on:
1. The BIOS half. The host firmware must copy its console to the serial port. Dell calls this Console Redirection, and on the R750xs it is controlled by BIOS settings with real names you can read over Redfish without rebooting: SerialComm (set to OnConRedirAuto on this machine) and RedirAfterBoot (set to Enabled). 2. The BMC half. The BMC watches the UART and wraps every byte into SOL payload packets inside the same RMCP+ session ipmitool already uses, on the same UDP port 623. No new firewall hole, no extra cable.
Follow one character of boot text left to right, then flip the redirection toggle and see what a half-configured SOL looks like.
{ "height": 280, "caption": "One character of boot text, host to terminal. Flip the BIOS half off.", "nodes": [ { "id": "host", "label": "Host boot text", "kind": "host", "x": 0, "y": 80, "detail": "Everything the host prints during boot: BIOS, GRUB, kernel. With redirection on, it all goes to the serial port too." }, { "id": "uart", "label": "ttyS1 (COM2)", "kind": "nic", "x": 240, "y": 80, "detail": "The serial port. BIOS Console Redirection copies the console here, character by character." }, { "id": "bmc", "label": "BMC SOL engine", "kind": "bmc", "x": 480, "y": 80, "detail": "Reads the UART and packs each byte into RMCP+ SOL payload packets." }, { "id": "mnet", "label": "Mgmt network", "kind": "net", "x": 720, "y": 80, "detail": "UDP port 623, the same session ipmitool already uses. Nothing new to firewall." }, { "id": "you", "label": "Your terminal", "kind": "admin", "x": 940, "y": 80, "detail": "sol activate renders the stream here, and your keystrokes ride the same pipe back to the host." } ], "edges": [ { "from": "host", "to": "uart", "label": "redirect", "kind": "plain" }, { "from": "uart", "to": "bmc", "label": "tap", "kind": "oob" }, { "from": "bmc", "to": "mnet", "label": "SOL payload", "kind": "mgmt" }, { "from": "mnet", "to": "you", "label": "your terminal", "kind": "mgmt" } ], "toggle": { "label": "BIOS Console Redirection:", "on": "Enabled (SerialComm=OnConRedirAuto)", "off": "Disabled", "dimOff": ["e:host-uart", "e:uart-bmc", "e:bmc-mnet", "e:mnet-you"] } }
With redirection off, the pipe still exists but nothing feeds it: your SOL session connects and shows a blank screen. That exact symptom is the number one SOL support call in the real world.
>>> The BMC half being on proves nothing about the feed: SOL is the pipe, BIOS redirection is the water. On real Dell gear your first stop is BIOS > Serial Communication, and on this machine you can read those same settings over Redfish without rebooting anything. A powered-off BMC would refuse the session, and a wrong cipher fails the login, not the picture.
Time to check both halves on the real work order, starting from the inside: the BMC's own shell. You opened this door in First Contact with sshpass -p ADMIN ssh -o StrictHostKeyChecking=no admin@$BMC_IP, which lands you at the /admin1-> prompt. The readout you want in there is one word: solconfig.
prompt: /admin1-> answer: solconfig output: Serial-Over-LAN Configuration ============================================================
SOL Status: Enabled Baud Rate: 115200 Minimum Privilege: admin Retry Count: 7 Retry Interval: 50 ms Payload Channel: 1 Payload Port: 623 hint: One word: solconfig
Read it like an operator:
The NOC does not open interactive shells for routine reads. The same facts are one command away from your workstation, over the network. IPMI calls these the SOL parameters and reads them per channel: channel 1, exactly as solconfig just told you.
prompt: ops@dc-east-ws01:~$ answer: ipmitool -I lanplus -C 3 -H $BMC_IP -U ADMIN -P ADMIN sol info 1 output: Set in progress : set-complete Enabled : true Force Encryption : false Force Authentication : false Privilege Level : ADMINISTRATOR Character Accumulate Level (ms) : 50 Character Send Threshold : 96 Retry Count : 7 Retry Interval (ms) : 50 Volatile Bit Rate (kbps) : 115.2 Non-Volatile Bit Rate (kbps) : 115.2 Payload Channel : 1 (0x01) Payload Port : 623 hint: Type it exactly: ipmitool -I lanplus -C 3 -H $BMC_IP -U ADMIN -P ADMIN sol info 1
Same machine, same facts, two dialects. solconfig is the firmware's view; sol info is the network's view. Map them: baud 115200 shows up here as a bit rate of 115.2 kbps, minimum privilege admin becomes Privilege Level ADMINISTRATOR, and retry 7 at 50 ms matches line for line. When the two views disagree on real gear, trust neither and investigate: something is rewriting the configuration behind your back.
>>> Garbled text is the classic baud mismatch signature: both ends are talking, just at different speeds, so every byte decodes wrong. Both ends must agree, and 115200 is the standard on modern Dell gear. A wrong password would have refused the session outright, and a bad cable gives you silence, not glyphs.
Reading the configuration is reconnaissance. The verb that matters is activate: the sol activate subcommand, on the same remote command form you just used, attaches your terminal to the machine's serial console, live. Whatever the host prints, you see. To capture evidence while you watch, pipe the session through tee, which copies the stream to your screen and to a file at the same time.
Once attached, you drive the session with the tilde escape ladder, the same idea SSH uses:
~? lists the escape commands~. detaches your terminal from the console (the host does not notice)~B sends a serial break to the hostOne more rule, learned the hard way at 03:00: a BMC seats exactly one active SOL session. If dayshift left a session attached, your activate is refused until the stale one is cleared, and clearing it is the same remote form ending in sol deactivate.
{ "height": 320, "caption": "The BMC seats exactly one SOL session. Clear the stale one and the seat is yours.", "nodes": [ { "id": "opsA", "label": "You (03:00)", "kind": "admin", "x": 0, "y": 60, "detail": "Night shift, trying to attach to the console right now." }, { "id": "opsB", "label": "Dayshift laptop", "kind": "admin", "x": 0, "y": 300, "detail": "Left a SOL session attached at 17:40 and went home. The BMC is still holding its seat." }, { "id": "bmc", "label": "BMC", "kind": "bmc", "x": 420, "y": 180, "detail": "Allows exactly ONE active SOL payload session at a time. A second activate is refused until the first is gone." }, { "id": "uart", "label": "ttyS1", "kind": "nic", "x": 660, "y": 180, "detail": "There is one serial port. One stream, one seat." }, { "id": "host", "label": "bay14-r750xs", "kind": "host", "x": 880, "y": 180, "detail": "Still stuck in GRUB, still printing to serial, indifferent to who is watching." } ], "edges": [ { "from": "opsA", "to": "bmc", "label": "sol activate", "kind": "oob" }, { "from": "opsB", "to": "bmc", "label": "stale session", "kind": "oob" }, { "from": "bmc", "to": "uart", "kind": "oob" }, { "from": "uart", "to": "host", "kind": "plain" } ], "toggle": { "label": "Stale session:", "on": "Still attached (you are refused)", "off": "After sol deactivate (you connect)", "dimOn": ["e:opsA-bmc"], "dimOff": ["opsB", "e:opsB-bmc"] } }
>>> Detaching from a console never touches power. SOL watches and types; it does not switch anything. The boot failure is still sitting there for the next session, which is exactly what you want while the OS team works the case.
The BMC in this lab is a standards-faithful emulator, and its IPMI stack does not carry a live SOL byte stream. The console you are about to open replays a captured boot of bay14-r750xs and closes itself when the capture ends. Every command, flag, field name, and escape sequence is exactly what you would use on a physical iDRAC.
Time to do it for real. Close Work Order DC-EAST-WO-1008 on the machine you are about to open. The BMC address is preset in $BMC_IP, credentials ADMIN / ADMIN.
Six objectives stand between you and going back to bed:
1. Confirm from the inside. Reach the controller's own shell and pull its Serial-over-LAN readout. 2. Read the line from outside. Query the SOL settings over the network, one shot, no interactive shell. 3. Open the console. Attach to the serial console and watch the boot attempt with your own eyes. 4. Capture the evidence. The NOC attaches console output to the OS team's case: land it in ~/boot-log.txt. 5. File the config. The next shift needs the SOL settings on record in ~/sol-config.txt. 6. The challenge. The OS team wants the exact failing line, the one that names the disk GRUB cannot find, in ~/root-cause.txt. No hints on this one.
Here is the deal: the workspace shows no commands. It hands you one objective at a time, and you recall the move. That recall is the whole point; it is how tonight's reading becomes your skill.
Stuck is normal. Hit Request a signal on any objective for a nudge, or ask Daemon. It knows exactly which task you are on and will nudge, not spoil.
Your workstation dc-east-ws01 is booting in the bay below, with the Dell BMC beside it on the management network. Somewhere behind it, bay14-r750xs is stuck in GRUB, printing its trouble to a serial port nobody is reading. Launch the workstation and be the one who reads it.
Practice Serial Over LAN in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.