Learn › Linux System Calls › Processes and Plumbing
strace -f -e trace=process bash -c "ls" - a hands-on Linux lab on a real virtual machine.
Follow a program as it becomes a second program: clone, execve, wait, and the reason fork shows up in a trace as clone.
Every time you type a command, something quietly remarkable happens. The shell you are typing into cannot turn itself into ls. A running program cannot swap out its own name and start over as a different one on a whim. So how does the shell run ls at all?
It does a three-step dance, and every program that starts another program does the same one. First it clones itself, making a second copy of the shell. Then that copy replaces its own program with ls. Then the original shell waits for ls to finish. Clone, replace, wait. That is how one program becomes another.
You are about to watch that dance happen, call by call. This lab has a tiny program called fork_demo that does exactly this: it makes a child, the child turns into echo, and the parent waits for it. You will trace it and see the three system calls with your own eyes.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every screen in this lesson was captured by booting this exact lab and running these exact commands. The one thing that will look different for you is the live trace: the child's process number and the long 0x... addresses change every single boot, and that is normal. The lab user is tracer and the machine is named syscall-lab. Your progress in the lab is tracked automatically, so type commands naturally.
Before you trace anything, just run the program and watch what it prints. fork_demo is already built and waiting in your home directory. Run it with ./fork_demo.
./fork_demo
prompt: tracer@syscall-lab:~$ answer: ./fork_demo ||| ~/fork_demo ||| /home/tracer/fork_demo output: child running parent done hint: The program is in your home directory. Run it with ./fork_demo
Two lines, from two different programs. child running was printed by echo, which the child turned into. parent done was printed by the original program after it waited for the child to finish.
That ordering is the whole story in miniature. The child ran first and printed its line. The parent held back until the child was done, then printed its own. One program made another, let it run, and collected it. Now you will trace the same run and see the exact system calls behind those two lines.
Here is the catch with tracing a program that makes a child. By default strace follows only the program you launched, the parent, and the child runs off untraced. To follow the child too, you add -f, which means follow forks.
Trace the demo with strace -f ./fork_demo. Because the trace lines carry process numbers and addresses that change every boot, read this as an example, not a screen to match. Yours will show different numbers.
execve("./fork_demo", ["./fork_demo"], 0x7ffe602d6e48 /* 10 vars */) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fc755f35a10) = 863
wait4(-1, strace: Process 863 attached
[pid 863] execve("/usr/local/bin/echo", ["echo", "child running"], 0x7ffdb28aeb08 /* 10 vars */) = -1 ENOENT (No such file or directory)
[pid 863] execve("/usr/bin/echo", ["echo", "child running"], 0x7ffdb28aeb08 /* 10 vars */) = 0
<... wait4 resumed>[{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 863Your PIDs and addresses will differ. The number 863 is the child's process id on that one boot; yours will be some other number, and it appears both after clone and inside the [pid 863] prefixes strace adds to mark the child's lines.
Three system calls carry the whole dance. clone makes the child. execve turns the child into echo. wait4 is the parent waiting for the child to finish. Read them in that order and the trace tells the same story your eyes saw a moment ago.
Take the three apart, because each one is a name worth knowing by sight.
clone is the call that brings a new process into being. It appears once, and its return value = 863 is the child's process id, handed back to the parent so the parent knows who to wait for.
execve is the call that loads a program. It replaces whatever program is running in a process with a new one. You see it twice near the top of the child's lines, which you will make sense of shortly.
wait4 is the parent pausing until the child is done. Notice it is split across two lines: wait4(-1, ... starts, then the child runs, then <... wait4 resumed>... = 863 finishes and hands back the same child id. The parent was asleep in wait4 the entire time the child was working.
This is the single most surprising fact in the lesson, so take a position before you read the answer.
>>> It is clone, and this is the headline of the whole lesson. The C code said fork(), and every book calls it fork, but there is no line named fork anywhere in the trace. The library's fork() is a thin wrapper that reaches the kernel as clone, and the trace shows the kernel's truth, not the program's wording. If you picked fork, you are in good company, almost everyone expects that line and it is not there. If you picked exec, that comes one beat later: execve is how the child replaces itself with a new program, but first the child has to exist, and clone is what brings it into being.
It is worth sitting with this, because it is the kind of gap between what code says and what the kernel does that trips people up for years.
On old systems there really was a separate fork call. Modern Linux replaced it with one flexible call, clone, that can make anything from a full separate process to a lightweight thread depending on the flags you hand it. When you call the plain old fork(), the C library calls clone for you with the exact flags that reproduce classic fork behavior.
You can even read those flags in the trace: SIGCHLD in the flags means the child sends its parent a signal when it exits, which is what makes an ordinary child process rather than a thread. So fork is not gone, it is a specific way of calling clone. The lesson: trace the kernel, not the manual, when you want to know what really ran.
The child began life as an exact copy of fork_demo. Then it called execve. Predict what that did to it.
>>> execve replaces. Up to that line the child was still running fork_demo's code, a clone of its parent. execve throws that program away and loads echo in its place, keeping the same process and the same process id. Nothing new is spawned by this line. If you picked the first answer, no second process appears here: the child you already made simply becomes echo. If you picked the third, the parent is never touched; execve only ever replaces the one process that calls it, which here is the child. Clone makes the process, execve fills it with a program. Two separate steps, and you can see both.
Look again at the child's two execve lines. The first one failed and the second one worked, and that is not a bug, it is the shell's habit made visible.
[pid 863] execve("/usr/local/bin/echo", ["echo", "child running"], ...) = -1 ENOENT (No such file or directory)
[pid 863] execve("/usr/bin/echo", ["echo", "child running"], ...) = 0When you run a bare command name like echo, the system does not know which directory holds it. So it walks the list of directories in your PATH, in order, trying execve on each until one succeeds. /usr/local/bin comes first and has no echo, so that execve returns -1 ENOENT, meaning no such file. /usr/bin/echo exists, so the next execve returns 0 and the child becomes echo.
You are watching a PATH search happen one call at a time. Every command you have ever typed by its short name did this same walk.
>>> It is the PATH search, and it is completely normal. Looking for a bare command means trying each directory in PATH until one has it, and /usr/local/bin simply does not, so that execve answers no with ENOENT. The very next line finds echo in /usr/bin and returns 0. If you picked the first answer, you would flag a healthy program as broken; a failed probe is just the answer no, and traces are full of them. If you picked permissions, the error would read EACCES instead. ENOENT is specific: it means there was nothing there to run, not that you were forbidden to run it.
One call is left, and it belongs to the parent. After clone returned the child's id, the parent called wait4 and went to sleep.
wait4 means: pause here until a child finishes, and tell me how it went. In the trace it starts on one line, the child does all its work while the parent sleeps, and then wait4 resumes and returns the child's id, 863, the same number clone handed back at the start. The WIFEXITED and WEXITSTATUS(s) == 0 inside it are the parent reading the result: the child exited, and its exit status was 0, meaning success.
This is why parent done printed last when you ran the program. The parent could not move on until wait4 came back. A parent that skips wait4 leaves a finished child lingering as a zombie, which is a story for another lesson. For now: clone makes it, execve fills it, wait4 collects it.
The natural way to imagine running a command is one clean step: the shell reaches out, grabs echo, and runs it. One action, one program appearing. The trace you just read is the proof that it does not work that way.
There is no single call that runs a command. There is clone, which makes a bare copy of the caller, an exact duplicate that is still the old program. Then there is execve inside that copy, which is what finally swaps in the new program. And there is wait4 in the original, holding until the copy is done. Three calls in two processes, not one call in one.
Once you see it this way, a lot of Linux stops being mysterious. A shell is a program that clones and execs, over and over, once for every command you type. When you understand these three calls, you understand the machinery under every command line you will ever use.
Your lab scores that you can name the three calls. Three short files hold them: the call that makes the child, the call that loads the new program, and the call that waits. Write each name into its file, then read all three back.
echo clone > answers/spawn_call.txt
echo execve > answers/loads_program.txt
echo wait4 > answers/waits.txtprompt: tracer@syscall-lab:~$ answer: echo clone > answers/spawn_call.txt; echo execve > answers/loads_program.txt; echo wait4 > answers/waits.txt; echo 'spawn:'; cat answers/spawn_call.txt; echo 'loads:'; cat answers/loads_program.txt; echo 'waits:'; cat answers/waits.txt ||| echo clone > answers/spawn_call.txt; echo execve > answers/loads_program.txt; echo wait4 > answers/waits.txt; echo spawn:; cat answers/spawn_call.txt; echo loads:; cat answers/loads_program.txt; echo waits:; cat answers/waits.txt output: spawn: clone loads: execve waits: wait4 hint: Echo each name into its file, then cat the three files back with a label before each
Three files, three facts. spawn_call.txt holds clone, the call that made the child. loads_program.txt holds execve, the call that turned the child into echo. waits.txt holds wait4, the call the parent used to collect it. The grader reads these files, so writing them is you stating, in order, the three steps of one program becoming another.
Stop and notice what changed. When you started, fork_demo printed two lines and that was all you knew. Now you can name every system call behind them. clone made a child and returned its id. execve walked PATH and replaced the child with echo. wait4 held the parent until the child was done. You followed all of it with one flag, -f.
That skill is not about fork_demo. Every shell, every service manager, every program that launches another program does this exact dance, and now you can trace any of them and read it call by call.
Scaffolding off. No command is printed from here on.
In the live lab, trace fork_demo with -f on your own and find the three calls in your own trace. Then look at the number clone returns and confirm it is the same number wait4 returns at the end. That number is the child's process id, and seeing it appear at both ends is seeing the parent recognize the exact child it made.
You can also trace a real shell running a command, strace -f -e trace=process bash -c "ls", and watch the same clone, execve, wait4 appear for an everyday command. The -e trace=process part filters the trace down to just the process calls so the three stand out.
You took the invisible thing that happens on every command line and read it, call by call. Here is every form this lesson taught:
| Command | What it does |
|---|---|
./fork_demo | Run the demo: a child runs echo, the parent waits |
strace -f ./fork_demo | Trace the demo and follow the child with -f |
strace -f -e trace=process bash -c "ls" | Trace a real shell spawning a command, process calls only |
echo clone > answers/spawn_call.txt | Record a call name as evidence for the grader |
And the facts underneath all of it:
clone makes the child and returns the child's process id to the parent.fork() reaches the kernel as clone. There is no line named fork.execve replaces a process's program in place, keeping the same process id.wait4 is the parent pausing until the child finishes and reading its exit status.When you trace anything that starts other programs, reach for -f first or you will follow only the parent and miss the children entirely. Pair it with -e trace=process to hide the memory and file setup and leave just clone, execve, wait4, and exit.
You saw execve load a program and wait4 read an exit status of 0. The next lessons open up what a process is made of: the descriptors it inherits, and how pipes and redirection wire one program's output into another's input.
This lesson walked you through one program becoming another. You ran fork_demo, followed its child with -f, and named the three calls: clone makes the child, execve loads the new program, wait4 collects it. You saw why the trace says clone where the code said fork, and why a bare command name causes a PATH search of failing execve calls until one succeeds.
The lab is that same work on a real machine with strace installed, fork_demo built, and an answers directory waiting. It scores five things: you followed the child with -f, you named the call that makes the child, you named execve, you named wait4, and you traced the demo with -f.
The objectives name the goal, not the command. You read what needs to be true, recall the form, and type it. Your progress is tracked automatically as you work, so type naturally, and run check-progress whenever you want to see your score.
Launch the lab and watch one program spawn another.
Practice Watching a Process Spawn in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.