Learn › Linux System Calls › The Boundary
strace -Z /home/tracer/broken_open - a hands-on Linux lab on a real virtual machine.
Read the errno on a failed syscall, find the one call that broke a program, and repair the environment until the trace comes back clean.
In the last lesson every system call you traced worked. Each one ended in a = 0 or a small number, the kernel saying yes. That is not how real work goes. Programs ask for files that were moved, directories they cannot enter, ports already taken. The kernel says no all day long.
So the useful question is not whether a call can fail. It is: when one does, how does the program find out, and how do you? The answer is one number and one short name. A failed system call returns -1, and it sets errno, a name that says exactly what went wrong.
On this machine two programs are planted broken. broken_open will not run. broken_perms will not run. Neither prints a stack trace or a wall of red. Each just quietly fails at one system call. By the end of this lesson you will trace each one, read the name it failed with, and fix the exact thing that name points at.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every screen in this lesson was captured by booting this exact lab and running these exact commands, and the failure lines are stable, so what you see here is what you will see. The lab user is tracer and the machine is named syscall-lab. Your progress in the lab is tracked automatically, so type commands naturally.
Start by looking at what you have. Two programs sit in your home directory, and one locked file they will need. List the programs, then look at the file that is locked shut.
ls broken_*
prompt: tracer@syscall-lab:~$ answer: ls broken_*; echo '--- secret is locked ---'; ls -l secret/data.txt ||| ls broken_* ; echo '--- secret is locked ---' ; ls -l secret/data.txt output: broken_open broken_perms --- secret is locked --- ---------- 1 tracer tracer 24 Aug 19 05:18 secret/data.txt hint: List the two programs and long-list the locked file: ls broken_*; ls -l secret/data.txt
Two programs, broken_open and broken_perms. And one file, secret/data.txt, whose permission column reads ----------. Every permission bit is off. Nobody can read it, not even you who owns it. Hold that; it is the second failure you will fix.
Right now you know something is wrong but not what. Running ./broken_open would just print a terse open: No such file or directory and stop. That is the program's own guess at the problem. You are about to get the kernel's exact answer instead, which is always more precise than the program's.
Trace broken_open the way you traced ls, but this time you are hunting for one thing: the call that broke. A failed call ends in = -1 followed by an errno name. Filter the trace for the file the program is reaching for.
strace ./broken_open 2>&1 | grep settings.conf
prompt: tracer@syscall-lab:~$ answer: strace ./broken_open 2>&1 | grep settings.conf ||| strace ./broken_open 2>&1 | grep config ||| strace ./broken_open 2>&1 | grep 'openat(' output: openat(AT_FDCWD, "/home/tracer/config/settings.conf", O_RDONLY) = -1 ENOENT (No such file or directory) hint: Trace the program and filter for the file it wants: strace ./broken_open 2>&1 | grep settings.conf
There is the whole story on one line. openat asked to open /home/tracer/config/settings.conf for reading, and the kernel handed back = -1 ENOENT.
Read the two parts of that answer. The -1 is the alarm: this call failed. The ENOENT is the diagnosis: it stands for Error, NO ENTry, which is the kernel's way of saying no such file or directory. The program asked for a file, and the file is not there. You did not have to guess. The trace named the file and named the reason.
Every system call answers the same way when it fails. It returns -1, and it sets a hidden variable called errno to a number. That number has a short name, and the name is the useful part.
Think of errno as the reason slot on a rejection notice. The kernel cannot hand you the file you asked for, so it hands back a slip that says -1 in big letters and, in small print, a code for why. ENOENT means the thing is not there. EACCES means you are not allowed. There are a few dozen of these names, but a working engineer meets the same handful over and over.
strace does the translation for you. Under the hood the kernel set the number 2, and strace printed ENOENT (No such file or directory) so you never have to memorize the numbers. You read the name, and the name tells you what to fix.
You have seen one failure up close. Before you meet the second, take a position on the pattern.
>>> It returns -1 and sets errno. That is the contract every system call keeps: success comes back as 0 or a useful number, and failure comes back as -1 with a name in errno. If you picked the first answer, nothing prints on its own, the program has to check the -1 and decide to say something, which is why a broken program can fail in silence. If you picked the third, the call itself does not halt anything, it just returns -1 and hands control back, and it is up to the program whether to stop or shrug and continue. The whole skill of this lesson is reading that -1 and the name beside it.
Now trace the second program. broken_perms reaches for that locked file you saw, secret/data.txt. It exists, so this will not be ENOENT. Trace it and read the name.
strace ./broken_perms 2>&1 | grep data.txt
prompt: tracer@syscall-lab:~$ answer: strace ./broken_perms 2>&1 | grep data.txt ||| strace ./broken_perms 2>&1 | grep secret ||| strace ./broken_perms 2>&1 | grep 'openat(' output: openat(AT_FDCWD, "/home/tracer/secret/data.txt", O_RDONLY) = -1 EACCES (Permission denied) hint: Trace the second program and filter for its file: strace ./broken_perms 2>&1 | grep data.txt
Same shape, different name. openat asked for /home/tracer/secret/data.txt, and this time the answer is = -1 EACCES (Permission denied).
The file is right there, so ENOENT would be wrong. EACCES means Error, ACCESS: you found the file, but you are not allowed to read it. Remember its permissions read ----------, every bit off. The kernel is enforcing that. ENOENT and EACCES are the two failures you will meet most, and they are opposites worth keeping straight: one says the thing is missing, the other says the thing is guarded.
>>> The file exists, and the program is not allowed to read it. EACCES is a permission wall, not a missing thing. If you picked the first answer, that is ENOENT, the other name, and telling the two apart is the entire point: a missing file and a forbidden file need completely different fixes. If you picked the third, a wrong name would also come back ENOENT, not EACCES, because the kernel would fail to find that path at all. EACCES only happens when the path resolves to a real file whose permissions say no.
So far you filtered with grep because you knew the file name. When you do not, modern strace has a flag that does the finding for you. -Z prints only the calls that failed, nothing that succeeded. Run it on broken_open.
strace -Z ./broken_open
prompt: tracer@syscall-lab:~$ answer: strace -Z ./broken_open ||| strace -Z ./broken_open 2>&1 output: access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/home/tracer/config/settings.conf", O_RDONLY) = -1 ENOENT (No such file or directory) open: No such file or directory hint: The -Z flag keeps only the failed calls: strace -Z ./broken_open
Out of the whole run, only two calls failed, and -Z shows just those two. No memory setup, no library opens, only the -1 lines. On a program that makes thousands of calls, this is how you find the needle without reading the haystack.
But look closely, because there is a trap here. Two calls failed, and they are not equal. The last line, open: No such file or directory, is the program's own message on its way out.
The trap is this: -Z printed two failures, and it is tempting to think both need fixing. They do not. Look at the top line.
access("/etc/ld.so.preload", R_OK) = -1 ENOENT. You have seen this exact line before. In the very first lesson, ls produced the identical failure, and you learned it was harmless: a program probing for an optional file that is not there, shrugging, and moving on. It failed here too, and it is just as harmless here.
The failure that matters is the second one, openat on settings.conf, because that is the file the program actually needs to do its job, and without it the program gives up. Two -1 lines, right next to each other, both ENOENT, and only one is a real problem. Telling the harmless failure from the fatal one is the whole skill. A -1 is not a verdict of broken; it is a fact you have to read in context.
>>> Only the openat for settings.conf. The ld.so.preload probe is the same optional check you met in lesson one, and it fails on a healthy machine every time. If you picked the first answer, you would go hunting for a file the system does not want and never miss, while the real problem sat one line down. If you picked the third, the trace is exactly right, it is faithfully reporting two failures, and your job is not to distrust it but to read which failure has consequences. That reading, harmless versus fatal, is what separates someone who can use a trace from someone who just stares at red text.
Now fix the one that matters. ENOENT told you the exact path the program wants: /home/tracer/config/settings.conf. So give it that. Create the directory, put a file there, and run broken_open again. Watch its exit code go from failure to 0.
mkdir -p config && echo ready > config/settings.conf
prompt: tracer@syscall-lab:~$ answer: mkdir -p config && echo ready > config/settings.conf; echo '--- run it again ---'; ./broken_open; echo "exit code: $?" ||| mkdir -p config; echo ready > config/settings.conf; echo '--- run it again ---'; ./broken_open; echo "exit code: $?" output: --- run it again --- ready exit code: 0 hint: Make the path the errno named, then rerun: mkdir -p config && echo ready > config/settings.conf; ./broken_open
The program runs. It opened the file it was missing, read the word ready, printed it, and exited 0, the code for success. You did not touch the program's code. You fixed its world. The ENOENT named a missing file, you supplied the file, and the failure was gone.
That is the shape of most real fixes. The trace tells you what the program needs, and you make that true. An exit code of 0 at the end is the program confirming it.
Your lab scores what you can show it found. Record the name of the failure you diagnosed. Write ENOENT, the errno on the call that broke broken_open, into the answers file, then read it back.
echo ENOENT > answers/errno.txt
prompt: tracer@syscall-lab:~$ answer: echo ENOENT > answers/errno.txt; cat answers/errno.txt ||| printf 'ENOENT\n' > answers/errno.txt; cat answers/errno.txt output: ENOENT hint: Echo the errno name into the answers file, then cat it back: echo ENOENT > answers/errno.txt
One word in a file, but it is the word that mattered. ENOENT is the name the kernel gave the failure, and writing it down is you stating the diagnosis the trace handed you. The grader reads this file, so recording it is the finding, not busywork.
Stop and notice what you can do now. Two programs came to you broken with no explanation. You traced each, found the single call that returned -1, read the errno beside it, and knew from the name alone what was wrong: one file missing, one file guarded. Then you fixed the missing one and watched the program run.
That skill does not stop at these two toy programs. Every failing program on a real machine fails at some system call, and every one of those failures returns -1 and sets a name. A web server that will not start, a script that cannot write its log, a backup that stalls: trace it, find the -1, read the name. You already know how.
Scaffolding off. No command is printed from here on.
One program is still broken: broken_perms. You already traced it and read its errno, EACCES, the file is there but locked shut with permissions ----------. This is a different fix from the last one. You are not creating anything. You are opening up a file that exists.
Add the read permission the owner needs on secret/data.txt, then run broken_perms and confirm it exits 0. The errno named the problem; make the thing it named true.
prompt: tracer@syscall-lab:~$ answer: chmod u+r secret/data.txt; echo '--- run it again ---'; ./broken_perms; echo "exit code: $?" ||| chmod +r secret/data.txt; echo '--- run it again ---'; ./broken_perms; echo "exit code: $?" ||| chmod u+rw secret/data.txt; echo '--- run it again ---'; ./broken_perms; echo "exit code: $?" output: --- run it again --- the secret is: penguins exit code: 0 hint: EACCES is a read-permission wall. Add the owner read bit, then rerun: chmod u+r secret/data.txt; ./broken_perms
The secret is penguins, and the program exits 0. EACCES said the file was guarded, you added the read bit the owner was missing, and the wall came down. A completely different repair from the missing-file fix, driven by a completely different errno, and you read which was which straight off the trace.
You took two programs that failed in silence and made them talk. Here is every form this lesson taught:
| Command | What it does |
|---|---|
strace ./broken_open | Run a failing program and watch for the call that returns -1 |
strace -Z ./broken_open | Print only the calls that failed, nothing that succeeded |
mkdir -p config && echo ok > config/settings.conf | Create the file an ENOENT names, so the program can find it |
chmod u+r secret/data.txt | Add the read bit an EACCES names, so the program can read it |
And the facts underneath all of it:
-1 and sets errno, a short name for what went wrong.strace prints the name for you, so you read ENOENT or EACCES, not raw numbers.ENOENT means the file is not there. EACCES means it is there but you are not allowed.-Z keeps only the failed calls, the fast way to find a failure in a long trace.-1 is not always a problem. A program probing for an optional file that is missing is healthy. Reading which failures matter is the skill.When a program fails and will not say why, reach for two things: strace -Z to isolate the failed calls, and the errno name on the one that matters. ENOENT sends you to create or find a path. EACCES sends you to permissions. The name is the map to the fix.
You have been reading errno names one at a time. Real programs check errno in their own code and change what they do based on it. That is how a good program turns a -1 into a clear message or a quiet retry instead of a silent death, and it is where this track goes next.
This lesson walked you through the whole loop. You traced two failing programs, found the -1 in each, and read the errno beside it. You used -Z to keep only the failures, and told the harmless probe from the fatal one. Then you repaired both: a file created for the ENOENT, a permission opened for the EACCES. You wrote the errno down as your finding.
The lab is that same work on a real machine, with the two broken programs waiting and an answers directory ready. It scores five things: you traced a failing program, you recorded the errno, you showed the failures with -Z, you repaired the missing file, and you repaired the permission.
The objectives name the goal, not the command. You read what needs to be true, recall the form, and type it. Your progress is tracked automatically as you work, so type naturally, and run check-progress whenever you want to see your score.
Launch the lab and fix what the errnos name.
Practice When System Calls Fail in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.