Learn › Linux Foundations › System Inspection
Final test: Chapter 21, System Inspection. Twenty-five objectives on a real Debian 12 server covering both parts: lscpu, nproc, lsblk, lspci and lshw, then free, ps, pgrep, top, lsof and kill, with five saved reports and two stops of report-worker by its process ID. Fifty minutes, graded from the commands you run, the reports you save and the log report-worker keeps.
This chapter covers the following topics:
This is Part 2 of 2. Part 1 covered Sections 21.1 to 21.5, the hardware Linux reports. This part covers Sections 21.6 to 21.11: memory, the processes running on the machine, the files they hold open, and how to end a process by its process ID. It ends in the chapter lab, which covers all eleven sections.
What you already know: Section 4.12 read /proc/meminfo, and Section 8.4 defined a process. Section 9.9 ended a job with kill and its job spec, and Section 10.10 printed process IDs with $ and $!. Section 11.1 numbered the three standard streams, Section 11.2 saved output with >, and Section 8.18 put the output of one command into the line of another with $( ). This part meets processes that are not jobs of your shell, finds them by name, and ends one by its number.
Objectives
After this part you can:
Select Start machine in the adjacent workspace and use its Terminal. The command cards show selected, recorded examples from a prepared server, with omitted lines marked ...; inspect the real output in your Terminal. Process IDs, times and memory figures change from moment to moment. The Section 21.11 exercises stop and restart report-worker on that real practice machine. The chapter assessment starts a separate fresh machine and checks commands, reports and the worker log. Follow the startup and session-time displays in the workspace.
Five questions before the material. A wrong answer costs nothing: the explanation states the fact and names the section that teaches it.
>>> free counts memory nothing uses at all. available estimates what a new program can get without swapping, including reclaimable cache. Section 21.6.
>>> Every running program is a process, and the kernel gives each one a number, its PID, which names it on the whole machine. A job number belongs to one shell only. Section 21.7.
>>> pgrep takes a name and prints the process ID of every process whose name matches. Section 21.8.
>>> lsof, list open files, prints every process that holds a given file open, and on which descriptor. Section 21.10.
>>> A job spec names a job of one shell, and only that shell knows it. A program the system started has no job number in your shell, and kill reaches it through its process ID. Section 21.11.
Memory, or RAM, holds the programs that are running and the data they work on, and it is emptied when the machine stops. Section 4.12 read the kernel's own report, /proc/meminfo, one figure per line. free prints the few figures from it that matter, in one table. It takes no path. Print it:
prompt: admin@server1:~$ answer: free output: total used free shared buff/cache available Mem: 983744 278048 387784 476 466732 705696 Swap: 507900 0 507900 hint: Section 21.6. The memory command free, with nothing after it.
Your figures differ, and they change from one moment to the next. The unit is the kibibyte, 1024 bytes, the kB of /proc/meminfo. The row Mem is memory. The row Swap is the swap disk sdb from Section 21.3.
Columns: total is all the memory the kernel can use. In this version of free, used is total minus available, not a sum of every process's memory. For example, 983744 minus 705696 is 278048 in the recorded row. Do not add used, free and buff/cache as if they were three separate pieces of total. free is memory nothing uses at all. shared includes memory shared by processes and some temporary filesystems. buff/cache includes cached file data and other memory the kernel can reclaim (Section 4.12). available is the kernel's estimate of what new programs can get without swapping; it is not simply free plus all of buff/cache.
-h prints every figure with a unit, the way df -h did in Section 5.11:
prompt: admin@server1:~$ answer: free -h ||| free --human output: total used free shared buff/cache available Mem: 960Mi 271Mi 378Mi 476Ki 455Mi 689Mi Swap: 495Mi 0B 495Mi hint: Section 21.6. The memory command with its human-readable option -h.
Ki, Mi and Gi are kibibytes, mebibytes and gibibytes, each 1024 times the one before, and B is bytes, so 0B is none. Swap used at 0B means the machine has not needed its swap disk. The readable form is shorter and less exact: use it to read, and the plain form when you need the exact figure.
Free memory can look small on a healthy server. Memory nothing uses does no work, so the kernel can use it to cache recently read files: buff/cache grows and free shrinks. When a program asks for memory, the kernel can reclaim part of that cache.
That is why available is often larger than free. It estimates the memory a new program can get without swapping. Read it alongside swap use and the program's behavior before deciding that the server needs more memory. A low free value alone does not justify a change.
Save the output of free -h into ~/inspect/answers/memory.txt, the way Section 21.2 saved the processor count:
prompt: admin@server1:~$ answer: free -h > ~/inspect/answers/memory.txt ||| free -h > inspect/answers/memory.txt output: hint: Section 21.6. free -h, then > and the path ~/inspect/answers/memory.txt (Section 11.2).
Silent; the table is in the file. Read it back with cat ~/inspect/answers/memory.txt:
total used free shared buff/cache available
Mem: 960Mi 158Mi 410Mi 540Ki 391Mi 801Mi
Swap: 495Mi 0B 495MiThe figures of the moment it was saved. free shows new ones every time it runs; the file keeps these.
Section 8.4 defined a process: one running program, with memory of its own. The kernel gives each process a process ID, or PID, a number that names it on the whole machine, and Section 10.10 printed two of them with $ and $!. Every process also records its current parent by the parent's PID, the PPID. A process can acquire a new parent if its original parent exits.
ps, process status, prints a snapshot of the process table: what is running at the moment you ask, and nothing after it. With no option it lists the processes of your own terminal. Run it:
prompt: admin@server1:~$ answer: ps output: PID TTY TIME CMD 3218 pts/0 00:00:00 bash 3219 pts/0 00:00:00 ps hint: Section 21.7. The process command ps, with nothing after it.
Your process IDs and terminal name differ. Two processes: bash, your shell, and ps itself, which was running while it took the snapshot. PID is the process ID. TTY is the terminal the process is attached to; pts/0 is a pseudo-terminal, the kind a remote session gets. TIME is the processor time the process has used so far, zero for anything this short. CMD is the name of the program.
The chapter directory holds a README about the process this part examines. Print ~/inspect/README.txt:
prompt: admin@server1:~$ answer: cat ~/inspect/README.txt ||| cat inspect/README.txt output: inspect: the working directory of Chapter 21 (System Inspection) ================================================================ answers/ empty when the machine boots. The chapter lab asks you to save some reports here. logs/worker.log the log of report-worker. It gains one line every five seconds while report-worker runs.
report-worker is a small program that runs in the background as admin from the moment the machine boots. It keeps logs/worker.log open for writing and, every five seconds, checks for reports to build and writes one line. It changes nothing else.
restart-worker starts report-worker again after you stop it, and does nothing while it is already running. hint: Section 5.1. The printing command, then the path ~/inspect/README.txt.
report-worker was started when the machine booted, by the system and not by your shell. It runs as admin, keeps logs/worker.log open, and writes one line to it every five seconds. restart-worker is a helper installed on server1, not a Linux command; Section 21.11 uses it after stopping report-worker.
-e selects every process on the machine, and -f prints the full format, with the owner, the parent and the whole command line. On this server the list runs to about a hundred lines. Print it:
prompt: admin@server1:~$ answer: ps -ef ||| ps -e -f output: UID PID PPID C STIME TTY TIME CMD root 1 0 2 15:12 ? 00:00:02 /sbin/init root 2 0 0 15:12 ? 00:00:00 [kthreadd] ... admin 2553 1 0 15:13 ? 00:00:00 /bin/bash /usr/local/bin/rep ... hint: Section 21.7. The process command with the options -ef.
Your list is longer and every number differs; the ... lines stand for the rows skipped here. Columns: UID is the account that owns the process. PID and PPID are its own number and its parent's. C is its share of a processor over its life, as a whole percent. STIME is when it started. TTY is its terminal, ? for none. TIME is the processor time used. CMD is the full command line.
Process 1 is /sbin/init, the name systemd is started under: the kernel starts it first, and it starts services and sessions. Process 2, kthreadd, belongs to the kernel, and names in square brackets are the kernel's own threads. The report-worker row reads: owned by admin, current parent 1, no terminal. Its command line is cut short: ps stops each line at the right edge of the terminal, and this listing was taken in a window 80 columns wide. In full it is /bin/bash /usr/local/bin/report-worker, and a wider window shows all of it. The fixture starts this worker through systemd at boot. A PPID of 1 alone would not prove who originally started an arbitrary process, because process 1 can adopt an orphaned process. The program is a bash script, so the command line names bash and then the script.
History. In 1984 Tom Killian described a filesystem that shows each running process as a file, in the paper Processes as Files, written for the Eighth Edition of Research Unix at Bell Labs. Linux took up the idea: /proc has held one directory per process since the early 1990s, and ps, top and pgrep read their answers from it.
ps accepts two families of options. Options with a dash come from System V Unix. Letters with no dash come from BSD, the Berkeley line of Unix, and ps aux is the BSD way to list every process, with memory figures added. Print it:
prompt: admin@server1:~$ answer: ps aux output: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 2.4 1.2 102436 12476 ? Ss 15:12 0:02 /sbin/init ... admin 2553 0.0 0.3 7072 3260 ? Ss 15:13 0:00 /bin/bash /us ... hint: Section 21.7. The process command with the three letters aux, written without a dash.
Your figures differ. New columns: %CPU and %MEM are the process's share of processor time over its life and of memory. VSZ is the memory it has mapped and RSS the part of it actually held in memory, both in kibibytes. START is when it started. STAT is its state. COMMAND is cut at the edge of the terminal, as in Exercise 21-19.
The first letter of STAT: S sleeping, waiting for something to happen; R running; I an idle kernel thread. Z is a zombie, a process that has finished but whose parent has not yet collected it. Letters after the first add detail: s the leader of a session, + in the foreground of its terminal, < high priority, N low priority, l several threads. report-worker is Ss: asleep between two lines of its log, and the leader of its own session.
-u and an account name select the processes that account runs. List the processes of admin:
prompt: admin@server1:~$ answer: ps -u admin ||| ps -U admin ||| ps --user admin output: PID TTY TIME CMD ... 3712 ? 00:00:00 report-worker ... hint: Section 21.7. The process command with -u and the account name admin.
Your list also holds the processes of your own session: your shell and the programs the system runs for your login. report-worker is admin's too. The columns are those of Exercise 21-17, and CMD is the short program name, not the whole command line.
-o names the columns to print, separated by commas and with no spaces: pid, ppid, stat and comm, the command name, here. With -u admin it prints admin's processes in those four columns:
prompt: admin@server1:~$ answer: ps -u admin -o pid,ppid,stat,comm ||| ps -o pid,ppid,stat,comm -u admin output: PID PPID STAT COMMAND ... 3712 1 Ss report-worker ... 4284 3712 S sleep ... hint: Section 21.7. ps -u admin, then -o and the column list pid,ppid,stat,comm.
Two rows shown. report-worker, whose parent is 1. And sleep, whose parent is report-worker's PID: the worker starts sleep between two lines of its log and waits for it to finish. That pair is the relation the PPID column records. -o takes any column name the ps manual lists, in any order, so a script prints exactly the columns it needs.
Reading the whole list to find one process is slow. pgrep, process grep, takes a name and prints the process ID of every process whose name matches it. Find report-worker:
prompt: admin@server1:~$ answer: pgrep report-worker output: 2553 hint: Section 21.8. The name search command pgrep, then the name report-worker.
One number, the process ID of report-worker. Yours differs: the kernel handed it out at boot. pgrep matches the process name, the short name in the CMD column of Exercise 21-21. A name that matches nothing prints nothing at all, and pgrep reports it only in its exit status, 1 (Section 10.3).
-a prints the whole command line beside each number, so you can see what you found:
prompt: admin@server1:~$ answer: pgrep -a report-worker output: 2553 /bin/bash /usr/local/bin/report-worker hint: Section 21.8. pgrep with its option -a, then the name report-worker.
The number, then the command line ps -ef showed in its CMD column. Before you act on a PID, check the current command, owner and intended effect. A name can match more than one process, and a PID from an earlier reading may later belong to a different process.
Save the process ID of report-worker into ~/inspect/answers/worker.pid:
prompt: admin@server1:~$ answer: pgrep report-worker > ~/inspect/answers/worker.pid ||| pgrep report-worker > inspect/answers/worker.pid output: hint: Section 21.8. pgrep report-worker, then > and the path ~/inspect/answers/worker.pid.
The number is in the file. Read it back with cat ~/inspect/answers/worker.pid:
3712A file that holds one process ID is a common pattern: many services write their own when they start, usually under /run, so that other programs can find them.
ps takes one snapshot. top takes a new one every three seconds and redraws the screen, the busiest processes first, under a summary of the whole machine. -b, batch mode, prints plain text instead of drawing a screen, and -n 1 stops after one update. That is the form scripts use, and it shows exactly what the screen holds:
prompt: admin@server1:~$ answer: top -b -n 1 ||| top -bn1 ||| top -b -n1 output: top - 15:14:06 up 1 min, 0 user, load average: 1.24, 0.52, 0.20 Tasks: 98 total, 1 running, 97 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 960.7 total, 378.7 free, 271.1 used, 456.3 buff/cache MiB Swap: 496.0 total, 496.0 free, 0.0 used. 689.6 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND ... 2553 admin 20 0 7072 3260 3004 S 0.0 0.3 0:00.00 report-+ ... hint: Section 21.9. top with -b for batch mode and -n 1 for a single update.
Your figures differ, and the ... lines skip most of the processes. Line 1 is the line of uptime from Section 1.10: the time, how long the machine has run, and the users logged in. It ends with the load average, the number of processes wanting a processor, averaged over the last 1, 5 and 15 minutes. Line 2 counts the processes by st
Practice Chapter 21, Part 2. System Inspection: free, ps, pgrep, top, lsof and kill in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.