Learn › BMC & Server Management › Serial Over LAN

Serial Over LAN

Final test: the whole Serial Over LAN chapter. Establish the state of an unreachable host from its controller, read the console configuration, find and clear a held console session, capture a real boot over the console to a file, convert it to readable text, extract the firmware handoff, the kernel banner and the line that explains the loss of network, record the console parameters this firmware refuses, file the genuine Dell console reference, and produce a written diagnosis. 25 objectives on a real workstation called mgmt with a real IPMI 2.0 controller at 10.100.0.2 owning a real virtual machine whose console you really read.

Chapter 7. Serial Over LAN

This chapter covers the following topics:

7.1 The serial console, and why it keeps working when nothing else does
7.2 Establishing the state of a host before touching its console
7.3 SOL configuration and what each parameter controls
7.4 The single-session payload, and releasing a session that holds it
7.5 Capturing a console session to a file
7.6 Reading a boot capture: firmware, kernel, userspace
7.7 Console parameters and the limits of this controller
7.8 Vendor implementations and their escape sequences
7.9 Recording what the console shows
7.10 Producing a diagnosis from console evidence

What you already know. Chapter 1 established the RMCP+ session and the -I lanplus -H -U -P -C 3 command form used throughout this track. Chapter 2 covered chassis power control, including the difference between a reset and a power cycle. Chapter 6 covered the channel, its sessions and its cipher suites. This chapter uses all three.

Objectives. After this chapter you can:

Explain what Serial over LAN carries, on which transport, and why it survives the loss of the host network
Read an SOL configuration and say what each parameter controls
Attach to and detach from a console, and reclaim one that is already in use
Capture a console session to a file and convert it to readable text
Identify the firmware, kernel and userspace phases in a boot capture
Produce a written diagnosis supported by console evidence

The machine. The workstation is a real Linode running Ubuntu 24.04. The controller beside it is a software IPMI 2.0 controller built on that machine at boot, managing a real virtual machine whose serial port is the far end of every console session in this chapter. Boot takes about 3 minutes. This chapter is free; the machine is yours for 60 minutes and stops itself after 20 minutes idle. The dark boxes below are practice terminals, one command each, graded as you type.

This chapter uses a real Linux workstation and software-emulated IPMI hardware. The protocol requests and replies are real; controller identity, sensor values and host behavior may be seeded or emulated. Supplied physical-hardware captures are labeled separately.

Pre-check 1 of 5

These five questions establish whether you need this chapter in full. Answer from what you know now. Each explanation names the section that covers the point.

[ ] A graphical console, rendered by the BMC and streamed as images
[x] The host serial port, relayed by the BMC inside an IPMI session
[ ] An SSH session to the host operating system
[ ] A copy of the host system log, read from disk by the BMC

>>> SOL relays the physical serial port of the host inside an IPMI session, on the same UDP port 623 as every other IPMI command. It is not graphical, it does not require an operating system, and it does not read anything from disk. Section 7.1.

Pre-check 2 of 5

[ ] The host is powered off
[ ] The BMC has failed over to a backup interface
[x] Nothing about the host by itself; the power state and sensors must be read to rule out hardware
[ ] The host operating system has panicked

>>> A responding BMC and an unresponsive host are consistent with many causes, including a powered-off host, a hardware fault, and a working host with no network. The power state and the sensor readings separate them, and they are the first two things to read. Section 7.2.

Pre-check 3 of 5

[x] It is refused until the occupied payload is released
[ ] It creates an independent console merely because a different username was used
[ ] It pre-empts the first operator automatically
[ ] It always opens another attachment regardless of payload state

>>> This lab has one available console payload instance. SOL implementations can expose multiple instances, and sharing limits are platform-specific. Read the actual refusal and coordinate with the current operator before releasing a console. Section 7.4.

Pre-check 4 of 5

[ ] ~. at any point in the line
[ ] Ctrl-C
[x] ~~. at the start of a line
[ ] exit

>>> The escape is a tilde followed by a dot and works only at the start of a line. ssh uses the same escape and consumes the first tilde, so inside an ssh session the console escape becomes two tildes and a dot. Using a single tilde there closes the ssh session instead. Section 7.8.

Pre-check 5 of 5

[ ] The file is binary because SOL is a binary protocol
[x] Null bytes, carriage returns and terminal escape sequences from the pty
[ ] The file is encrypted by the RMCP+ cipher suite
[ ] Nothing; script writes only a summary unless told otherwise

>>> script records everything the pseudo-terminal carried, which includes null bytes, carriage returns and the escape sequences the far end used to control the display. The capture is converted to text with tr and sed before it is read. Section 7.5.

7.1 The serial console, and why it survives

A serial console is a text terminal attached to a machine over a serial port. Before a machine has a network, before it has drivers, and before it has an operating system, it has a serial port, and firmware writes to it.

Serial over LAN carries that serial port inside an IPMI session. The BMC sits on the far end of the physical port, reads what the host writes, and relays it to you over the same UDP port 623 that every other command in this track has used. Keystrokes travel the other way.

The consequence is the reason SOL exists. A serial console does not need the network stack of the host, its NIC driver, or an operating system that has finished booting. It works at a firmware prompt, after a kernel panic, while a filesystem check is running, and when the host NIC is down. It is the last access path to stop working, which makes it the first one to reach for when a machine goes quiet.

The path is workstation, management network, BMC, then the host serial port. A failed host NIC does not break that path, but the management connection and configured serial output still have to work.

With the link down, every path through the NIC disappears and the serial path is untouched, because it never depended on any part of the network stack. That is the whole argument for SOL, and it is also the condition of the host in this chapter.

One limitation belongs here rather than later. SOL carries a serial console. If firmware or the operating system was never told to write to the serial port, the session opens and shows nothing. A silent SOL session is far more often a missing console= kernel argument or an unset firmware option than a broken controller.

7.1.1 The access path

The controller is reached exactly as in every previous chapter. -I lanplus selects the encrypted RMCP+ interface, -H the controller address, -U and -P the account, -C 3 the cipher suite.

Objective 1 records the controller identity. Standing in your home directory, run ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 mc info > ~/console/mc-info.txt:

prompt: ops@mgmt:~$ answer: ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 mc info > ~/console/mc-info.txt ||| ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 mc info > /home/ops/console/mc-info.txt ||| ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 mc info > console/mc-info.txt output: hint: The same mc info used in every chapter, redirected into mc-info.txt.

Silent: the output went to the file. Two lines in it matter for this chapter. IPMI Version : 2.0 is a precondition, because SOL is an IPMI 2.0 payload and does not exist in IPMI 1.5. Firmware Revision identifies the implementation, which determines which of the optional SOL parameters are present, as section 7.7 shows.

7.2 Establishing the state of the host

The host does not respond to ssh or to ICMP. Before opening a console, establish whether the host is running at all, because the answer changes what the console is expected to show.

Objective 2. Standing in your home directory, run ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 chassis power status > ~/console/power.txt:

prompt: ops@mgmt:~$ answer: ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 chassis power status > ~/console/power.txt ||| ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 chassis power status > /home/ops/console/power.txt ||| ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 chassis power status > console/power.txt output: hint: chassis power status reports the host power state. Redirect it into power.txt.

Silent. cat ~/console/power.txt prints:

Chassis Power is on

The host is powered. That eliminates the simplest explanation and it also sets an expectation for the console: a powered host that is not on the network should still be producing output on its serial port, or should at least respond to a reset with a boot.

Note the exact wording. chassis power status reports Chassis Power is on. The chassis status command reports the same fact as System Power : on inside a longer block. They are different commands with different output, and a script that greps for one will not find the other.

7.2.1 Ruling out a hardware fault

Goal: read the reported sensors before opening the console. These measurements help establish context; they are not a complete hardware or OS health test.

prompt: ops@mgmt:~$ answer: ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sensor list > ~/console/sensors.txt ||| ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sensor list > /home/ops/console/sensors.txt ||| ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sensor list > console/sensors.txt output: hint: The sensor table from chapter 4, redirected into sensors.txt.

The recorded readings are within their configured limits. Power draw and fan motion are consistent with a powered system but do not prove the CPU is executing useful work.

Check: keep these values as supporting evidence, then use the console to see how far boot progresses and what errors appear. Do not eliminate all hardware faults from this table alone.

7.3 SOL configuration

sol info reports how the console is configured on a channel. Objective 4. Standing in your home directory, run ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sol info 1 > ~/console/sol-info.txt:

prompt: ops@mgmt:~$ answer: ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sol info 1 > ~/console/sol-info.txt ||| ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sol info 1 > /home/ops/console/sol-info.txt ||| ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sol info 1 > console/sol-info.txt output: hint: sol info takes a channel number. Redirect channel 1 into sol-info.txt.

cat ~/console/sol-info.txt prints:

Info: SOL parameter 'Authentication (2)' not supported
Info: SOL parameter 'Character Interval (3)' not supported
Info: SOL parameter 'Retry (4)' not supported
Info: SOL parameter 'Payload Channel (7)' not supported - defaulting to 0x01
Info: SOL parameter 'Payload Port (8)' not supported - defaulting to 623
Set in progress                 : set-complete
Enabled                         : true
Force Encryption                : false
Force Authentication            : false
Privilege Level                 : Unknown (0x00)
Character Accumulate Level (ms) : 0
Character Send Threshold        : 0
Retry Count                     : 0
Retry Interval (ms)             : 0
Volatile Bit Rate (kbps)        : IPMI-Over-Serial-Setting
Non-Volatile Bit Rate (kbps)    : IPMI-Over-Serial-Setting
Payload Channel                 : 1 (0x01)
Payload Port                    : 623

The Info lines report parameters this request could not retrieve. Some following fields may therefore be ipmitool fallback values. Do not treat an unsupported parameter's displayed default as a measured security policy. Check the platform documentation and the actual session configuration.

The fields that matter:

Field	What it controls
`Enabled`	Whether the channel will carry a console at all. `false` means an attach is refused regardless of credentials.
`Force Encryption`	Whether the controller refuses to carry the session in the clear. A valid false value means encryption is not required by that setting, not that the active session is necessarily unencrypted. Here the Authentication parameter was unsupported, so this field is not verified policy.
`Force Authentication`	Whether an unauthenticated payload is accepted.
`Privilege Level`	The account level required to open a console. Unknown (0x00) here is not a usable access decision. Verify the actual platform's supported policy and enforcement.
`Character Accumulate Level`	How long the BMC waits to fill a packet before sending it.
`Character Send Threshold`	How many characters trigger an immediate send.
`Retry Count`, `Retry Interval`	How hard the BMC retries an unacknowledged packet.
`Volatile` and `Non-Volatile Bit Rate`	The serial line speed, until reboot and across reboots.
`Payload Channel`, `Payload Port`	Where the payload is carried: channel 1, UDP 623.

The accumulate, threshold and retry values are the flow control of a serial stream over an unreliable network. They are zero on this controller and are real numbers on the vendor capture in section 7.8. When a console feels laggy or drops characters on a busy link, those are the parameters to look at.

7.4 The single-session payload

This lab offers one SOL console payload instance. While it is occupied, another attach to that instance is refused. Other implementations can support multiple instances or different sharing rules; inspect the actual platform.

A session is established on this controller now. Objective 5 attempts an attach and records the result.

Two things about the command. ipmitool requires a pseudo-terminal for a console session; without one it fails on tcgetattr and streams nothing, so the attach runs under script -qec, which allocates a pty and records the session. timeout 12 bounds it, and < /dev/null keeps script from consuming the rest of whatever is feeding it.

Standing in your home directory, run the command below:

prompt: ops@mgmt:~$ answer: timeout 12 script -qec 'ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sol activate' /dev/null < /dev/null > ~/console/blocked.txt 2>&1 output: hint: Run sol activate under script -qec with a timeout, stdin from /dev/null, and redirect both streams into blocked.txt.

grep -a "already active" ~/console/blocked.txt prints:

Info: SOL payload already active on another session

Info: SOL payload already active on another session. The command exits non-zero and nothing else happens.

Read what the message does and does not say. It reports state: a session exists. It is not an authentication failure and not a permission failure, and immediate retries do not resolve an occupied payload. An owner may disconnect or a configured timeout may later release it; coordinate before intervening. The grep -a is there because the file also contains what script recorded, including control bytes.

7.4.1 Enumerating sessions

Before releasing a payload, establish what is holding it. Objective 6. Standing in your home directory, run ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 session info all > ~/console/sessions.txt:

prompt: ops@mgmt:~$ answer: ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 session info all > ~/console/sessions.txt ||| ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 session info all > /home/ops/console/sessions.txt output: hint: session info takes a selector. Ask it for all, redirected into sessions.txt.

Silent. The file is long: this controller walks all 63 of its session slots and prints a block for each, most of them empty. The rows that carry information look like this:

session handle                : 1
slot count                    : 63
active sessions               : 2
user id                       : 2
privilege level               : ADMINISTRATOR
session type                  : IPMIv2/RMCP+
channel number                : 0x01

...

Each populated block reports a session handle, the user id that opened it, the privilege level it holds, the session type and the channel. Two sessions are present: the one holding the console, and the one running this command.

The same count is available in one line from the channel, which is usually what you want in a script:

  Active Session Count  : 2

7.4.2 What an unclean detach leaves behind

Before releasing the payload, one behaviour to understand, because it will affect you twice in this chapter.

A clean ~. escape tells the controller to release the SOL payload. If the client disappears without a clean detach, this lab's controller can still show that payload as active and refuse another attach.

If that happens, sol deactivate releases the old payload so you can attach again. Read the response rather than assuming every interrupted client leaves the same state.

A dropped connection may not release the payload immediately. The practical rule in this lab is: if a new attach is refused because the old payload is still active, use sol deactivate, then try again. Section 7.5 practices that recovery. Real controllers can also expire sessions after a timeout.

7.4.3 Releasing the payload

Objective 7. sol deactivate instructs the controller to drop the current payload. Standing in your home directory, run ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sol deactivate > ~/console/deactivate.txt 2>&1:

prompt: ops@mgmt:~$ answer: ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sol deactivate > ~/console/deactivate.txt 2>&1 ||| ipmitool -I lanplus -H 10.100.0.2 -U ADMIN -P ADMIN -C 3 sol deactivate > /home/ops/console/deactivate.txt 2>&1 output: hint: sol deactivate releases the payload. Redirect it with 2>&1 so a refusal is captured as well as a success.

Silent on success, which is why the redirect carries 2>&1: the interesting case is the one where it has nothing to release. Running it a second time prints:

Info: SOL payload already de-activated
(the second clear says it is already done)

Info: SOL payload already de-activated. That distinction is worth keeping: the first call released a session, the second told you there was nothing to release. In a runbook those are different outcomes.

Use the command deliberately. It takes the console from whoever holds it, which may be another operator working the same incident. Enumerate the sessions first and say what you are about to

Practice Serial Over LAN in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.