Learn › Linux System Calls › Diagnosis and the Modern Kernel
strace -o /home/tracer/date.trace date - a hands-on Linux lab on a real virtual machine.
Find out why some calls never appear in strace, the difference between the call you write and the call the kernel sees, and the tool that shows the hidden ones.
In the last lessons strace was a window with no blind spots. Every time a program reached outside itself, you saw the slip go past. That was the whole promise: nothing a program does to the outside world is hidden from the trace.
That promise has a hole in it, and this lesson is about finding it. Run date and it prints the time. Reading the clock is the most ordinary thing a program can ask for. So trace date, look for the call that asks the kernel what time it is, and you will not find one. Not because you missed it. Because it was never made.
By the end of this lesson you will know why the most common calls on the whole system never show in an strace, the one tool that can see them, and two more places where the name you write is not the name the kernel hears.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every screen in this lesson was captured by booting this exact lab and running these exact commands. Your own addresses and PIDs will be your own. The lab user is tracer and the machine is named syscall-lab. Your progress in the lab is tracked automatically, so type commands naturally.
Save a trace of date to a file, the clean way you learned, then search it for the calls that read the clock. Their names are gettimeofday and clock_gettime. Count how many times either one appears.
strace -o date.trace date
prompt: tracer@syscall-lab:~$ answer: strace -o date.trace date >/dev/null; echo 'time syscalls found in the trace:'; grep -cE 'gettimeofday|clock_gettime' date.trace; echo '(zero: the kernel was never asked)' ||| strace -o date.trace date; grep -cE 'gettimeofday|clock_gettime' date.trace output: time syscalls found in the trace: 0 (zero: the kernel was never asked) hint: Save the trace with -o, then count the time calls: grep -cE 'gettimeofday|clock_gettime' date.trace
Zero. A program whose entire job is to print the time made no system call to ask for the time.
This is not date being clever, and it is not strace being broken. The trace is complete. Every real system call date made is in that file, and none of them read the clock. The clock read still happened, your machine printed the right time. It just happened somewhere strace cannot look.
The rest of this lesson is about that somewhere.
Here is the trick the kernel plays. Some calls are made so constantly, by nearly every program, that going through the front door for each one would be wasteful. Reading the clock is the clearest example. A busy program can ask for the time thousands of times a second.
So the kernel maps a tiny shared library into the memory of every process when it starts. It is called the vDSO, the virtual dynamic shared object. Inside it sits the code for a handful of the most common calls, including the clock reads. When your program calls clock_gettime, it runs that code right there in its own memory and gets the answer back. It never crosses into the kernel, so there is no system call, so strace has nothing to see.
Think of it as the kernel leaving a small answer card taped inside every program's own room. For the questions on the card, the program does not have to walk to the service window at all. It reads the card and moves on.
Take a position before you go looking for the call.
>>> In user space, inside the vDSO. The kernel put that code in your program's own memory, so the call runs without ever crossing into kernel mode, which is exactly why the trace is empty. If you picked the first answer, the trace really is complete: nothing was filtered, the call simply never became a system call to record. If you picked the third, there is real hardware behind the clock, but your program does not talk to it; the vDSO code does the reading on the kernel's behalf, in your program's memory.
If the clock read never becomes a system call, strace will never show it, no matter what flags you use. strace watches the border between a program and the kernel, and this call does not cross that border.
There is a second tracer that watches a different line. ltrace traces library calls: the functions a program calls inside shared libraries like the C library, one layer above the kernel. The clock read is a library function before it is anything else, so ltrace sees it plainly. Run ltrace date and among the output is the line that strace could not give you:
date->clock_gettime(0, 0x7ffc2db50390, 0, 0x559d923ea2f0) = 0Read it as a sentence. The date program called clock_gettime, a function in the library, and it returned 0 for success. The long 0x... values are memory addresses, and they will be different every time you run this, which is normal and nothing to memorize.
So the call was there the whole time. strace looks at the kernel border and sees nothing, because the vDSO answered before the border. ltrace looks at the library border and sees the call in the open.
>>> Library calls. ltrace sits at the border between your program and the shared libraries it loads, which is one layer above where strace sits. That higher border is where the vDSO clock read is visible, because it is a library function even when it never reaches the kernel. If you picked the first answer, the two tools watch different lines on purpose: strace the kernel border, ltrace the library border, and this whole lesson lives in the gap between them. If you picked the third, that is a different idea entirely; ltrace shows the calls a program makes into libraries, passed and failed alike.
It is natural to walk out of the first lessons believing that strace shows everything a program does. That belief is useful, and it is also not quite true, and the empty date trace is the proof.
strace shows everything a program asks the kernel to do. That is a narrower promise, and it is the exact promise. Anything a program does inside its own memory, including running vDSO code, including plain arithmetic, is invisible to strace because none of it is a system call. The tool is not incomplete. It is precise about what it watches.
Hold both tools in mind and the picture is whole again. strace for the kernel border, ltrace for the library border. When a call goes missing from one, you now know to look at the other.
The vDSO is the sharpest version of a pattern that runs all through tracing: the call a programmer writes and the call the kernel records are often not the same name. You met this once already. Back in your first trace, the code almost certainly called open to open a file, but every line in the trace said openat. The library takes the friendly open you wrote and makes the more general openat call the modern kernel actually provides.
That is the rule stated plainly. You write the library call. strace shows the real kernel call. The two have a fixed relationship, and part of reading traces is knowing the common swaps by sight. open becomes openat. And there is one more swap, bigger than either, waiting in the next step.
When a program makes a copy of itself to run a second task, the classic call is fork. Generations of code call fork. But trace a program that forks and you will not see the word fork at all. You will see clone.
The lab has a small program, fork_demo, that calls fork and then runs echo in the copy. Trace it with strace ./fork_demo and the line that makes the new process looks like this:
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f4dc69a1a10) = 877The C library turns the old, simple fork into clone, a single more general call that can make a new process or a new thread depending on its flags. The = 877 at the end is the process ID of the new child, and the 0x... address, like the PID, will be different every time you run it.
So fork becomes clone, exactly as open becomes openat. Same rule, bigger swap.
>>> It shows clone. The library takes the fork in your source and calls clone, the one flexible call the kernel offers for making both new processes and new threads. If you picked the first answer, this is the same lesson openat taught: the trace records the kernel's call, not the programmer's wording. If you picked the third, making a new process is very much the kernel's job, one of its heaviest, so it is a real system call every time; only the clock read hides in the vDSO, not this.
Your lab scores what you can show it found. Three one-word files capture the three truths from this lesson: why the time call was invisible, the real name behind open, and the real name behind fork.
echo vdso > answers/why_invisible.txt
echo openat > answers/open_becomes.txt
echo clone > answers/fork_becomes.txtprompt: tracer@syscall-lab:~$ answer: echo vdso > answers/why_invisible.txt; echo openat > answers/open_becomes.txt; echo clone > answers/fork_becomes.txt; echo 'why_invisible:'; cat answers/why_invisible.txt; echo 'open_becomes:'; cat answers/open_becomes.txt; echo 'fork_becomes:'; cat answers/fork_becomes.txt ||| cat answers/why_invisible.txt answers/open_becomes.txt answers/fork_becomes.txt output: why_invisible: vdso open_becomes: openat fork_becomes: clone hint: Echo one word into each file, then cat all three back with a label before each
Three files, three facts. why_invisible.txt says vdso, the library the kernel maps in so the clock read never becomes a system call. open_becomes.txt says openat, and fork_becomes.txt says clone, the two swaps between the name you write and the name the kernel sees. The grader reads these, so writing them is not busywork; it is you stating what the traces told you.
Scaffolding off. No command is printed from here on.
You wrote clone into the answer file on the strength of what you read here. Now go see it with your own eyes. In the lab, trace fork_demo and find the clone line for yourself, the one that ends with the child's process ID.
It is the same move you have made all lesson: put a tracer in front of a program and read what it really does. The trace will not say fork anywhere. When you find clone, you have confirmed the last fact from the inside instead of taking it on faith.
Stop and notice what changed. An hour ago strace looked like it showed everything. Now you know its exact edge: it shows the kernel border, and the most common calls on the system, the clock reads, slip past that border through the vDSO and never appear. You know the one tool that catches them, and you know two places where the name in the code is not the name in the trace.
That is the difference between running a tracer and reading one. A missing call no longer stumps you. You ask which border it would cross, point the right tool at that border, and there it is.
You took the tool you trusted and found its blind spot, then found the tool that sees into it. Here is every form this lesson taught:
| Command | What it does |
|---|---|
strace -o date.trace date | Trace date and save every kernel call to a file |
grep clock_gettime date.trace | Search the trace for the time call, and find nothing |
ltrace date | Trace library calls, and see the time call strace could not |
strace ./fork_demo | Watch fork show up under its real name, clone |
And the four facts underneath all of it:
strace shows what a program asks the kernel to do, not everything it does.ltrace watches the library border, one layer above the kernel, so it sees calls strace cannot.open becomes openat, and fork becomes clone.When a call you expected is missing from an strace, do not assume the program skipped it. Ask whether it ran in the vDSO, and reach for ltrace to check the library border. A missing kernel call is often a present library call.
You have been reading traces as text. Real programs make so many calls that the text scrolls faster than you can read. The next lessons turn to counting and filtering at scale: summaries, following a running program by its PID, and pulling the one call you care about out of thousands.
This lesson walked you through the whole surprise. You traced date and found no time call, learned that the vDSO runs it in user space, used ltrace to see the call strace could not, and met the two swaps where open becomes openat and fork becomes clone. Then you wrote down the three facts the traces told you.
The lab is that same work on a real machine with strace and ltrace installed, an answers directory waiting, and a fork_demo program ready to trace. It scores five things: you traced date and captured it, you used ltrace to see the hidden call, and you recorded the three facts about the vDSO, openat, and clone.
The objectives name the goal, not the command. You read what needs to be true, recall the form, and type it. Your progress is tracked automatically as you work, so type naturally, and run check-progress whenever you want to see your score.
Launch the lab and find the call that was never there.
Practice The Invisible System Calls in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.