Learn › Linux System Calls › Diagnosis and the Modern Kernel
strace /home/tracer/mystery - a hands-on Linux lab on a real virtual machine.
A mystery program will not run. Using only tracing, find what it looks for, where it fails, and fix the box until it succeeds.
A program landed on this machine. It is called mystery. Someone ran it, it printed one line, and it quit. There is no source code, no README, no note about what it needs. It just does not work.
This is the situation every Linux engineer ends up in sooner or later. A program will not start, and its own error message is not enough to tell you why. You cannot read the code, or reading it would take an hour you do not have.
There is a faster way, and you have been building it this whole track. You put the program under strace and read what it asks the kernel for. The trace does not care whether the program has good error messages. It shows every system call the program makes and exactly which one fails. That is the whole job today: read the trace, find the failing call, fix what it needs, and prove the program runs.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every screen in this lesson was captured by booting this exact lab and running these exact commands. The lab user is tracer and the machine is named syscall-lab. The program is at ~/mystery. Your progress in the lab is tracked automatically, so type commands naturally.
Start where anyone would. Run the program and read its complaint. ./mystery prints a single line and exits:
open /home/tracer/etc/app.conf: No such file or directoryThat is more than many programs give you, and it is a real clue. But it is only as good as the message its author happened to write. Plenty of programs print error and nothing else, or print the wrong thing, or print nothing at all and just exit. You cannot build a skill on hoping the author was kind.
So you do not trust the message. You confirm it, and you learn to find the same fact for a program that gives you no message. Both come from the trace.
Put strace in front of the program. It runs mystery and prints every system call along the way. You are looking for the one call that comes back with -1, because that is where the program first asked the kernel for something and did not get it.
Run the program to see it fail, then trace it and pull out the failing open. strace ./mystery 2>&1 | grep 'openat' filters the live trace down to the open call.
prompt: tracer@syscall-lab:~$ answer: ./mystery; echo "exit code: $?"; echo '--- the trace names the missing file ---'; strace ./mystery 2>&1 | grep 'openat' ||| ./mystery; echo "exit code: $?"; echo '--- the trace names the missing file ---'; strace ./mystery 2>&1 | grep ENOENT output: open /home/tracer/etc/app.conf: No such file or directory exit code: 1 --- the trace names the missing file --- openat(AT_FDCWD, "/home/tracer/etc/app.conf", O_RDONLY) = -1 ENOENT (No such file or directory) hint: Run ./mystery to see it fail, then strace ./mystery 2>&1 | grep 'openat' to find the call that returns -1
Read the failing line slowly: openat(AT_FDCWD, "/home/tracer/etc/app.conf", O_RDONLY) = -1 ENOENT.
The program asked the kernel to open /home/tracer/etc/app.conf for reading. The kernel answered -1, which is how every system call says it failed, and then named the reason: ENOENT. You met ENOENT in the errno lesson. It means no such file or directory. The file the program needs is simply not there.
Notice this is the same openat from the very first lesson, the call any program uses to open a file. When it succeeds it hands back a small number, a file descriptor. When it fails it hands back -1 and an errno. The trace made the program's problem exact: not something is wrong, but this one path does not exist.
Step back and name what you just did, because it is the real skill and it works on any program on the machine.
A program that will not start is asking the kernel for something it cannot get. The trace is the record of every ask and every answer. You read down it until you find the first -1, you read the call name to see what the program was trying to do, and you read the errno to see why it could not. Failing call, plus errno, equals the cause.
That is forensics. You did not read the code. You did not guess. You watched the program at the one window where it talks to the kernel, and the window told you exactly what it was missing. Everything else today is acting on what the trace already told you.
The failing line ended = -1 ENOENT. The -1 is the return value, and ENOENT is the errno, the kernel's short name for the reason. Take a position on what that name is saying before you act on it.
>>> ENOENT means no such file or directory: the path is not there at all. If you picked permission denied, that is a different errno, EACCES, and it matters because the fix is different: EACCES means the file exists and you change its permissions, while ENOENT means you create the file. If you picked disk full, that is ENOSPC, another distinct errno. The reason the errno name is worth reading precisely is that each one points at a different repair. Here it is ENOENT, so the file is missing and your job is to make it.
A real trace can be hundreds of lines. When a program will not run, you need a rule for where to look, not a habit of reading all of it.
-1, because that is where the program first could not get what it asked for.mmap and brk lines, because most failures are memory problems.>>> You look for the first call that returns -1. A program asks for things in order, and the first ask it cannot satisfy is usually the thing it is actually missing. If you picked the last line, a failing program often stops well before its end, so the last line you see may just be the exit call after the real problem already happened. If you picked the address lines, mmap and brk are ordinary setup that almost always succeeds, and their addresses are noise that changes every run. Scan the return values, stop at the first -1, and read that line.
Your lab scores what you can show it found. Two short files capture the two facts the trace gave you: the path the program needs, and the errno it fails with. The grader reads these files, so writing them is you stating the finding plainly.
echo /home/tracer/etc/app.conf > answers/needs_file.txt
echo ENOENT > answers/fails_with.txtprompt: tracer@syscall-lab:~$ answer: echo /home/tracer/etc/app.conf > answers/needs_file.txt; echo ENOENT > answers/fails_with.txt; cat answers/needs_file.txt answers/fails_with.txt ||| echo /home/tracer/etc/app.conf > answers/needs_file.txt; echo ENOENT > answers/fails_with.txt; cat answers/needs_file.txt; cat answers/fails_with.txt output: /home/tracer/etc/app.conf ENOENT hint: Write the path into answers/needs_file.txt and the errno into answers/fails_with.txt, then cat both back
Two files, two facts. needs_file.txt holds the exact path the trace showed the program opening, and fails_with.txt holds the errno that open failed with. You read both straight off the one failing line: openat(AT_FDCWD, "/home/tracer/etc/app.conf", O_RDONLY) = -1 ENOENT. Path on the left, errno on the right. That single line is the entire diagnosis.
When a program fails, the first instinct is often to blame the program: it is broken, it was built wrong, reinstall it, find a newer version. Hold that instinct up against what the trace actually showed.
The trace showed a healthy program doing exactly what it was written to do. It tried to open a config file, the file was not there, and it reported the problem and stopped. Nothing about the program is broken. What is broken is the environment around it: the box is missing a file the program expects.
This is the shift the whole track has been driving at. A program is not a thing that acts on the world on its own. It asks the kernel for what it needs, and when one of those asks comes back no, the fix is usually out in the environment, not inside the program. You do not repair mystery. You give it the file it asked for.
The trace told you the program opens /home/tracer/etc/app.conf for reading. So create it. The etc directory does not exist yet either, so make the directory and the file in one step.
The program reads whatever the file contains and writes it back out prefixed with ok: . Give the config a real line so the result is readable.
mkdir -p etc && echo "greeting=hello" > etc/app.confThat is the entire repair. No change to the program, no reinstall. You created the one file the trace proved it was missing.
You have created the file. Before you run the program again, predict what the same openat line in the trace will look like this time.
-1 ENOENT again, because the program remembered the file was missing.0, because a successful system call always returns 0.>>> It returns a small non-negative number, a file descriptor. Programs do not remember a past failure; each run asks the kernel fresh, and this time the file is there, so the open succeeds and hands back a descriptor like 3. If you picked -1 ENOENT again, that was the answer only while the file was missing, and you fixed that. If you picked 0, be careful: openat returns a file descriptor on success, not 0. It is calls like close that return 0 for success. A successful open gives you a number you can go on to read from.
Now run the program with the file in place. It should read the config, succeed, and leave its output in out/result.txt. Run it, check the exit code, and read the file it wrote.
prompt: tracer@syscall-lab:~$ answer: echo '--- run it again ---'; ./mystery; echo "exit code: $?"; echo '--- the output it wrote ---'; cat out/result.txt ||| echo '--- run it again ---'; ./mystery; echo "exit code: $?"; echo '--- the output it wrote ---'; cat ~/out/result.txt output: --- run it again --- exit code: 0 --- the output it wrote --- ok: greeting=hello hint: Run ./mystery, echo its exit code, then cat out/result.txt to see what it produced
The exit code is 0, which means success, and the program wrote ok: greeting=hello into out/result.txt. That greeting=hello is the line you put in app.conf, read by the program and written back out with ok: in front. The whole loop closed: the program opened the config you created, read it, and produced its result.
You fixed a program you never read. You watched it fail, read the one call that failed, learned it wanted a file that was not there, created the file, and watched it succeed. That is the entire method, start to finish.
Stop and notice what you just did. You were handed a program with no source and no documentation, and you made it work. You did it with one tool and one habit: put it under strace, and read down to the first call that returns -1.
Everything in this track fed into that. Reading a trace line came from the first lesson. Knowing that openat returns a file descriptor on success came from the descriptors lesson. Reading ENOENT as no such file came from the errno lesson. The capstone did not teach a new trick. It showed you that the tricks you already have add up to a real skill.
Scaffolding off. Nothing here is graded, and no command is printed. This is for you.
Pick any command on the box that you expect to fail, and trace it to find out why. Try cat /etc/nope, or ls /root, or a program run against a file that does not exist. Trace it, scan the return values, and find the first -1. Read the call name and the errno, and say out loud what the program was trying to do and why the kernel said no.
You will find the errno is often different from ENOENT. A file you are not allowed to read fails with EACCES, not ENOENT. Each errno points at a different repair, and you now know how to read which one you are looking at.
You took a broken program with no source and made it run, using only the trace. Here is the method as a set of moves:
| Move | What it does | |
|---|---|---|
strace ./mystery | Run the program under the tracer and watch every system call | |
| `strace ./mystery 2>&1 \ | grep 'openat'` | Filter the trace down to the open calls to find the failing one |
Read the first -1 | The first failed call is usually the thing the program is missing | |
| Read the errno | ENOENT means create the file, EACCES means fix a permission | |
| Fix the environment | Give the program the file or permission the trace proved it needs |
And the facts underneath the method:
-1 and an errno. The errno names the reason and points at the fix.openat returns a file descriptor on success and -1 on failure.When a program will not start and you are not sure why, do not open the source first. Put it under strace, scan the return column for the first -1, and read that line. Nine times out of ten it names a file, a permission, or a directory that is not where the program expected it.
This is the last lab in the track. You started by watching ls ask the kernel for things, and you finish by using that same skill to repair a program you had never seen. The window between a program and the kernel is open to you now, on any machine you touch.
This lesson walked the whole method. You ran the mystery program and watched it fail, traced it and found the openat that returned -1 ENOENT, recorded the missing path and the errno, created the file the trace proved it needed, and ran the program again to watch it succeed and write ok: greeting=hello.
The lab is that same work on a real machine, with the mystery program compiled and waiting and an answers directory ready. It scores five things: you traced the program, you recorded the missing input file, you recorded the errno it fails on, you repaired the environment, and you proved the program now runs.
The objectives name the goal, not the command. You read what needs to be true, recall the move, and type it. Your progress is tracked automatically as you work, so type naturally, and run check-progress whenever you want to see your score.
Launch the lab and make the program run.
Practice Syscall Forensics in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.