Learn › RHCSA (EX200) › Operate Running Systems
journalctl - a hands-on Linux lab on a real virtual machine.
Read the systemd journal with journalctl and make it survive reboots. Filter by unit with -u, by severity with -p err, by time with --since, and follow live with -f; then the exam move: the journal is volatile by default, so create /var/log/journal and restart systemd-journald to make it persistent, after which --list-boots and -b -1 can recall previous boots. Serves EX200 operating-systems task 16: preserve system journals.
A service crashed overnight. You arrive, log in, and go straight to the logs to see what happened right before it died. You run journalctl and scroll back, but the trail stops cold at this morning's boot. Everything from before the reboot is simply gone. The evidence you needed erased itself the moment the machine restarted.
That is not a bug. On a fresh RHEL 10 machine the systemd journal is volatile: it lives in memory and is wiped on every reboot. Fixing that, making the journal survive reboots, is EX200 operating-systems task 16: preserve system journals. By the end of this lesson you will read the journal like an engineer and then flip it to persistent so the next crash leaves a trail you can follow.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. The outputs shown are from a real RHEL 10 machine (AlmaLinux 10.2). Journal timestamps, the hostname, and the boot count are live values, so yours will read differently. What stays true everywhere is the SHAPE of each command and the volatile-by-default behavior, and those are the lesson.
systemd-journald is the logging service that ships with RHEL. It collects messages from the kernel, from every service systemd starts, and from applications, and stores them in one structured binary journal. You never read that binary directly. You read it through one tool: journalctl.
Run journalctl with no arguments and it prints every entry it holds, oldest first, in a pager. Because that is a firehose, the real skill is filtering: by which service, by how severe, by when. That filtering is most of what the exam tests, so it is where we spend our reps.
For decades Linux logs were plain text files under /var/log, one per service, each in its own format, read with cat and grep. systemd replaced that patchwork with a single indexed journal so you can slice all logs by service, priority, or time with one tool instead of guessing which file to open.
The old text files did not disappear. rsyslog still writes /var/log/messages and friends alongside the journal, and that is its own lesson. journalctl is the systemd-native way in, and it is the one the exam leans on for filtering and for the persistence task.
The first filter you reach for narrows the journal to a single unit, a single service. The flag is -u, for unit, followed by the service name. Ask the journal to show only what the SSH server logged, and you get login records and nothing else.
Before you run it, decide what you expect: sshd handles logins, so its lines should mention accepted logins and opened sessions. Run the filter:
journalctl -u sshd
prompt: student@servera:~$ answer: journalctl -u sshd output: Jul 05 19:19:13 servera.lab.example.net sshd-session[1843]: Accepted password for student from 10.0.2.2 port 45168 ssh2 Jul 05 19:19:13 servera.lab.example.net sshd-session[1843]: pam_unix(sshd:session): session opened for user student(uid=1000) by student(uid=0) hint: The unit filter is -u followed by the service name: journalctl -u sshd
Every line is from sshd and nothing else. The -u sshd told the journal to drop all the kernel noise, all the other services, and hand you just this one unit's story: a password accepted, then a session opened. That is the move you make when a specific service misbehaves. Name the unit, read its lines. Swap in any service name, journalctl -u chronyd, and you get that service's records the same way.
Those exact timestamps, the hostname servera.lab.example.net, and the process ID 1843 belong to the machine this was captured on. Yours will differ every login. What is stable is the FORM: journalctl -u <service> returns only that service's lines, whatever they say today.
When you are hunting a problem, you do not want every chatty info line. You want the trouble. The -p flag filters by priority, the severity of a message, and it shows the level you name AND everything more severe. So -p err gives you errors, and above errors sit crit, alert, and emerg, while it hides the calmer warning, notice, info, and debug.
The priority names, most severe first, are: emerg, alert, crit, err, warning, notice, info, debug. Ask for errors and up:
journalctl -p err
prompt: student@servera:~$ answer: journalctl -p err output: hint: The priority flag is -p, then the level name err: journalctl -p err
On a healthy machine this can come back nearly empty, which is itself good news: nothing at error level or worse. The point is the CUT. -p err keeps err, crit, alert, emerg and throws away everything gentler, so you read only what might actually be breaking. Remember the direction: -p err is err AND ABOVE. It does NOT include warning. If you want warnings too, name the lower level: -p warning.
The priority direction trips people up on exam day. -p err means err and every level more severe, never less. If the task says 'show warnings and errors', you must use -p warning, because that level and up sweeps in err, crit, and the rest. Naming err alone would silently drop the warnings.
Two more filters finish your reading kit. --since limits the journal to a time window, and it takes plain-English phrases as long as you quote them: journalctl --since "1 hour ago" or journalctl --since "09:00". The quotes matter, because without them the shell splits 1 hour ago into three arguments and journalctl chokes.
The other is -f, for follow. journalctl -f behaves like tail -f: it prints the newest entries and then stays open, streaming each new line as it arrives, until you press Ctrl-c. It is what you leave running in one window while you reproduce a bug in another.
Combine an hour window with a single unit to see one service's recent story:
journalctl -u sshd --since "1 hour ago"
prompt: student@servera:~$ answer: journalctl -u sshd --since "1 hour ago" output: Jul 05 19:19:13 servera.lab.example.net sshd-session[1843]: Accepted password for student from 10.0.2.2 port 45168 ssh2 Jul 05 19:19:13 servera.lab.example.net sshd-session[1843]: pam_unix(sshd:session): session opened for user student(uid=1000) by student(uid=0) hint: Stack the filters: -u sshd for the unit, then --since with a quoted time phrase: journalctl -u sshd --since "1 hour ago"
The filters stacked. -u sshd narrowed to one service and --since "1 hour ago" narrowed to a time window, and together they returned only sshd lines from the last hour. That is the pattern for every journal query you will write: pick a unit, pick a severity, pick a window, and combine as many as you need. Each flag makes the result smaller and sharper.
Here is the heart of task 16. On a fresh RHEL 10 machine the journal is written to /run/log/journal, and /run is a memory-backed filesystem that empties on every reboot. So today's logs exist, but the moment the machine restarts they are gone. That is why your crash trail vanished in the cold open.
There is one probe that tells you which mode you are in: does the directory /var/log/journal exist? If it does, the journal is persistent and lives on disk. If it does not, the journal is volatile. Check it:
test -d /var/log/journal && echo persistent || echo "volatile (runtime only)"
prompt: student@servera:~$ answer: test -d /var/log/journal && echo persistent || echo "volatile (runtime only)" output: volatile (runtime only) hint: Test for the directory, print persistent on success or the volatile message on failure: test -d /var/log/journal && echo persistent || echo "volatile (runtime only)"
volatile (runtime only). The directory /var/log/journal is absent, so the journal is running from memory and every reboot wipes it. This is the default state of a fresh RHEL 10 install, and it is exactly the state the exam hands you when it asks you to preserve journals. The fix is to create that directory and tell journald to use it, which is the next step.
Making the journal persistent is two commands. First you create the on-disk directory that journald looks for: sudo mkdir -p /var/log/journal. The instant that directory exists, journald's default Storage=auto setting will start writing there. Second, you restart the service so it picks up the change now instead of at the next boot: sudo systemctl restart systemd-journald.
After that, re-run the same probe you ran a moment ago and it flips its answer.
The precise files and boot history on a real machine differ, but the sequence is fixed: create /var/log/journal, then restart systemd-journald. An alternative to relying on the default is to set Storage=persistent in /etc/systemd/journald.conf and restart, which forces persistence regardless of the directory. Both land the same result; creating the directory is the shorter path most engineers use on the exam.
Persistence earns you something concrete: history across reboots. journalctl --list-boots lists every boot the journal has recorded, one line each, and journalctl -b -1 then replays the logs from the previous boot. On a volatile journal there is only ever one boot to show, because the last reboot erased the rest. Once the journal is persistent, the list grows every time the machine restarts.
Count the boots the journal currently knows about:
journalctl --list-boots | wc -l
prompt: student@servera:~$ answer: journalctl --list-boots | wc -l output: 2 hint: List the boots, then pipe to wc -l to count the lines: journalctl --list-boots | wc -l
2. The journal has records from two boots, which means previous-boot logs are retrievable: journalctl -b -1 would replay the boot before this one, and journalctl -b 0 the current one. That number climbs with every reboot once the journal is persistent, and it is the proof the exam grader looks for. A volatile journal would show only 1, because the reboot took the rest with it. Your own count will differ, but the move is the same: list the boots, and each one you can revisit.
Scaffolding off. No command is printed this time. You have every piece you need.
A verification task wants you to confirm the journal is now persistent by checking for the on-disk journal directory and printing persistent when it is there. Think about which test operator asks whether a directory exists, and which logic operator runs the echo only when that test is true. Keep it to a single line, and target the standard persistent-journal path.
prompt: student@servera:~$ answer: test -d /var/log/journal && echo persistent output: hint: The directory test is -d, the path is /var/log/journal, and && runs echo only on success. Build the test, then && echo persistent.
The -d /var/log/journal test asks whether the persistent-journal directory exists. If you have created it, the test succeeds, && lets the echo run, and you see persistent. If the directory is still absent, the test fails, && swallows the echo, and the line prints nothing, the honest report that the journal is still volatile. That single line is task 16's checkpoint: one probe that tells the truth about whether your logs will survive the next reboot.
You earned this cheat sheet. Every row is a form you just ran or built:
The one thing to burn in for the exam: the journal is volatile by default, so 'preserve system journals' means create /var/log/journal and restart systemd-journald. Do it early in the exam as insurance, before any task asks you to read a previous boot.
journalctl -p err shows err AND above, never below. To include warnings you must ask for -p warning. And journalctl -b -1 needs persistence to work: on a volatile journal there is no previous boot to replay, which is exactly why the persistence task comes first.
The practice terminal has shown you the shape of the journal: journalctl to read it, -u to narrow to one service, -p err to keep only the severe, --since for a time window, -f to follow live, --list-boots to see boot history, and the two-command move that flips the journal from volatile to persistent so those boots survive. Every one of those you typed yourself.
The operating-systems module mission is where you run these against a real RHEL 10 machine. A full VM boots for you, with its own live journal and its own default-volatile state waiting to be fixed. The mission hands you objectives that use exactly what you practiced here: read a service's logs, filter by priority and time, and preserve the system journal so a previous boot can be recalled. One difference from this lesson: the mission shows no commands. You read the objective, recall the journalctl filter or the mkdir and restart, and type it. That recall is what makes it stick on exam day.
Finish the other operating-systems lessons, then go make a real machine's logs outlive a reboot.
Practice journald: Persistent Journals in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.