Learn › Linux System Calls › The Boundary
strace ls - a hands-on Linux lab on a real virtual machine.
Run strace on a program you already know and watch it talk to the kernel.
Here is a fact that sounds wrong the first time you hear it. The ls program cannot read a directory. It cannot put a single character on your screen. It cannot open a file. On its own, a program can shuffle numbers around inside its own memory and nothing else.
Everything that reaches outside the program, the disk, the screen, the network, the clock, is done by the kernel, and a program gets to it by asking. Each ask is a system call. ls does not list your files. It asks the kernel to, and then asks the kernel to print the answer.
You are about to watch those asks happen. strace sits between a program and the kernel and prints every request as it goes past. By the end of this lesson you will run strace on a command you use every day and read, line by line, what it is really doing.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every screen in this lesson was captured by booting this exact lab and running these exact commands. Your own addresses and byte counts will be your own. The lab user is tracer and the machine is named syscall-lab. Your progress in the lab is tracked automatically, so type commands naturally.
The simplest way to use strace is to put it in front of a command. Everything after it is the program to run and watch. Run strace ls and look at the very top of what comes back, before ls has listed anything at all. It looks something like this:
execve("/usr/bin/ls", ["ls"], 0x7ffc34cfa170 /* 10 vars */) = 0
brk(NULL) = 0x5560292d8000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1b14c0e000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=19967, ...}, AT_EMPTY_PATH) = 0
close(3) = 0Every line is one system call. The name comes first, the arguments are in the parentheses, and after the = is what the kernel handed back.
Read the first line. execve("/usr/bin/ls", ...) is the call that loaded the ls program in the first place, and it returned 0, meaning it worked. So the very first thing you learn is that even starting a program is a system call.
The long strings like 0x5560292d8000 are memory addresses, and they will be different every single time you run this, which is normal and nothing to memorize. You have not even reached your files yet: every line here is ls getting ready, asking for a little memory and opening the loader cache. A program is far more than the code someone wrote in it, and the trace shows the whole of it.
Look past the memory setup and one call keeps appearing: openat. It is how a program opens a file. Pull just those lines out of the trace with grep.
strace ls 2>&1 | grep 'openat('
prompt: tracer@syscall-lab:~$ answer: strace ls 2>&1 | grep 'openat(' ||| strace ls 2>&1 | grep openat ||| strace ls 2>&1 | grep -m3 'openat(' output: openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 hint: Filter the live trace for the open call with grep: strace ls 2>&1 | grep 'openat('
Read one line: openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3. It asks to open the C library for reading, and the kernel hands back 3.
That 3 is the important part. When you open a file, the kernel does not give you the file. It gives you a number, and from then on you use the number to talk about that file. That number is a file descriptor, the whole subject of the next lesson. For now, notice that the answer to openat is always a small number.
You also learned why the call is openat and not open. The program's code almost certainly said open, but the kernel's real call is openat, and the trace shows the kernel's truth, not the program's wording. Hold that thought; it comes back.
The kernel is the part of Linux that owns the hardware. Your program runs in user mode, walled off from the disk and the memory of every other program. The kernel runs in kernel mode, where it can touch anything. A system call is the one doorway between the two.
The picture worth keeping is a service window. Your program cannot go into the back room where the hardware is. It writes a request on a slip, rings the bell, and waits. The kernel reads the slip, does the work in the back, and hands the result out through the window. strace is standing at the window reading every slip.
That is the whole idea, and everything else in this track is learning to read the slips. There are only a few hundred system calls, and a handful come up again and again: opening files, reading and writing them, starting programs, asking for memory. You will know the common ones by sight before long.
You are about to find the write that prints the file list. Take a position first.
>>> It is 1. Every program starts life with three descriptors already open: 0 is standard input, 1 is standard output, 2 is standard error. A write to 1 is a program putting something on your screen. If you picked 0, that is the one programs read from, not write to, and it is the easiest of the three to swap in your memory, so fix it now. If you picked the third answer, these three are not chosen by the program, the shell hands them over already open, which is why every program can just print without opening anything first.
A trace printed straight to the screen mixes with the program's own output. Send the trace to a file instead with -o, and the two stay apart. Then look inside the file for the write to descriptor 1.
strace -o ls.trace ls
prompt: tracer@syscall-lab:~$ answer: strace -o ls.trace ls >/dev/null; echo 'saved, line count:'; wc -l < ls.trace; echo '--- the write line inside it ---'; grep 'write(1' ls.trace ||| strace -o ls.trace ls; grep 'write(1' ls.trace output: saved, line count: 67 --- the write line inside it --- write(1, "START-HERE.txt\nanswers\nls.trace\n", 32) = 32 hint: Save with -o, then grep the file for the write to descriptor 1: strace -o ls.trace ls; grep 'write(1' ls.trace
There it is: write(1, "START-HERE.txt\nanswers\nls.trace\n", 32) = 32.
Read it as a sentence. Write, to descriptor 1, these bytes, and there are 32 of them. The kernel wrote all 32 and returned 32 to say so. That one line is the entire visible output of ls. Everything you see when you run ls is this single system call.
The \n marks are newlines inside the string, which is why the three names land on three lines on your screen. And the whole trace was 67 calls to list three files. Almost all of it was setup; one write did the part you actually see.
When you ran the very first strace ls without -o, that write line looked messier, because strace's report and ls's real output landed on the same line with nothing between them. Sending the trace to a file with -o is what makes it read cleanly.
One trace of ls was 67 lines. Real programs make thousands of calls, too many to read one at a time. strace -c runs the program and prints a summary instead of the live trace. Predict what it counts.
>>> One row per kind of call, with a count. The summary collapses the whole run into a table: openat was called five times, write once, and so on. If you picked the first answer, the point of -c is to stop reading calls one at a time and see the shape of the run instead. If you picked the third, that is a different and genuinely useful flag you meet in the errors lesson; -c counts everything, passed and failed alike, and the failures show up in their own column.
One trace of ls was 67 lines. Real programs make thousands of calls, too many to read one at a time. strace -c ls runs the program and prints a summary instead of the live trace. ls lists your files as usual, and then a table like this prints underneath:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
0.00 0.000000 0 5 openat
0.00 0.000000 0 1 write
0.00 0.000000 0 2 getdents64
0.00 0.000000 0 2 2 access
0.00 0.000000 0 1 1 ioctl
...Every row is one kind of call. The calls column is how many times it happened, and the errors column counts the ones that failed. Your own table will list the same calls with the same counts, but the rows may be in a different order and the time columns will show your machine's own numbers, because timing changes run to run.
Two rows have errors and both are fine. access failed twice, looking for files that are not there, which is normal probing. ioctl failed once because ls asked whether descriptor 1 is a terminal and got told no. Neither is a problem; a failed call is often just a program checking.
Look at getdents64, called twice. That is the call that actually reads directory entries, the real work of ls. You could not spot it in the top of the trace because it happens near the end, after all the setup. The summary surfaces it at a glance. When someone asks why a program is slow, this table is the first place you look.
It is natural to imagine ls reaching into the directory itself, walking the files, and printing them. Almost every beginner pictures it that way, and the trace you just read is the proof that it is wrong.
Nowhere in that trace did ls touch a disk. It asked the kernel to open things and got back numbers. It asked the kernel to read directory entries with getdents64. It asked the kernel to write the result with write. ls never left its own memory. It filled out slips and read the answers.
This is the mental shift the whole track rests on. A program is not a thing that acts on the world. It is a thing that asks, and the kernel acts. Once you see programs this way, strace stops being a debugging trick and becomes a way to read what any program on the machine is truly doing.
>>> No, and this is worth being firm about because it changes how you read every trace from here on. ls asked whether an optional preload file exists. It does not, so the call returned -1 with the name ENOENT, which means no such file. ls shrugged and moved on. Programs probe for optional files constantly, and each probe that comes back no is a normal part of a healthy run. If you picked the first answer, you would flag half of every real trace as broken. If you picked the third, whether the program reacts does not change what happened: the call was made and it failed, and the trace records both. Learning to tell a harmless failure from the one that matters is the entire errors lesson, which is next.
Your lab scores what you can show it found. Two short files capture the two facts from this lesson. You already have ls.trace saved, so pull the write line straight out of it, and name the call that opens a file.
grep 'write(1' ls.trace > answers/the_write.txt
echo openat > answers/opens_a_file.txtprompt: tracer@syscall-lab:~$ answer: grep 'write(1' ls.trace > answers/the_write.txt; echo openat > answers/opens_a_file.txt; echo 'the_write.txt:'; cat answers/the_write.txt; echo 'opens_a_file.txt:'; cat answers/opens_a_file.txt ||| cat answers/the_write.txt answers/opens_a_file.txt output: the_write.txt: write(1, "START-HERE.txt\nanswers\nls.trace\n", 32) = 32 opens_a_file.txt: openat hint: Redirect the write line into one file and echo openat into the other, then cat both back
Two files, two facts. the_write.txt holds the exact system call that prints ls's output, and opens_a_file.txt names the call that opens a file. The grader reads these, so writing them is not busywork; it is you stating what the trace told you.
Stop and notice what changed. An hour ago ls was a command that listed files. Now it is a program that starts with execve, asks for memory, opens libraries and gets back numbers, reads directory entries with getdents64, and prints the result with one write to descriptor 1. You can see all of it.
That skill does not stop at ls. The same strace reads a web server, a database, a program that will not start. Every one of them is asking the kernel for things, and you now know how to watch.
Scaffolding off. No command is printed from here on.
Trace whoami instead of ls, and find the single write that puts your username on the screen. It is the same shape as the one you found in ls: a write to descriptor 1.
Save the trace to a file first, the way you learned, so the write line reads cleanly.
prompt: tracer@syscall-lab:~$ answer: strace -o who.trace whoami; grep 'write(1' who.trace ||| strace -o who.trace whoami >/dev/null; grep 'write(1' who.trace ||| strace -o who.trace whoami; grep write who.trace output: write(1, "tracer\n", 7) = 7 hint: Same idea as before, saved to a file: strace -o who.trace whoami; grep 'write(1' who.trace
write(1, "tracer\n", 7) = 7. Seven bytes: the six letters of tracer and one newline. A completely different program, the identical pattern. Every program that prints does it with a write to descriptor 1, and now you can find that line in any of them.
You took a command you have run a thousand times and read what it actually does. Here is every form this lesson taught:
| Command | What it does | |
|---|---|---|
strace ls | Run a command and print every system call it makes | |
| `strace ls 2>&1 \ | grep 'openat('` | Filter a live trace for one kind of call |
strace -o ls.trace ls | Send the trace to a file so it does not mix with output | |
strace -c ls | Print a summary table counting each kind of call |
And the five facts underneath all of it:
= and the return value.1 is standard output.write to descriptor 1.When a trace scrolls past too fast, reach for two things first: -o file to save it so you can read it slowly, and -c to see the shape of the run before you read any single call. Almost every real investigation starts with one of those two.
You saw openat return numbers like 3 and never looked at what those numbers are for. That is the next lesson: file descriptors, the small integers every open file is known by, and the three every program is born with.
This lesson walked you through the whole loop. You ran strace on a command, read the calls, filtered for openat, saved a trace with -o, found the write to descriptor 1, and counted the run with -c. Then you wrote down the two facts the trace told you.
The lab is that same work on a real machine with strace already installed and an answers directory waiting. It scores five things: you traced a command, you saved a trace to a file, you found the write that prints output, you ran the -c summary, and you named the call that opens a file.
The objectives name the goal, not the command. You read what needs to be true, recall the form, and type it. Your progress is tracked automatically as you work, so type naturally, and run check-progress whenever you want to see your score.
Launch the lab and read what ls is really doing.
Practice Your First System Call in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.