Learn › Linux System Calls › Diagnosis and the Modern Kernel
sudo strace -p PID - a hands-on Linux lab on a real virtual machine.
Attach strace to a program that is already running, read what a hung process is blocked on, and use the summary to find where the time goes.
Everything you have traced so far, you started yourself. You put strace in front of a command and watched it run from the first call to the last. Real problems rarely arrive that way.
The usual shape is this. A program is already running. It has been running for hours. And it is stuck, or slow, and nobody restarted it because restarting it would lose whatever it is in the middle of. You cannot go back and put strace in front of it. It is already going.
So you attach. strace can walk up to a process that is already running, tap it on the shoulder, and start printing its system calls from that moment on. In this lesson you will attach to a stuck program and read, in one line, exactly what it is waiting for. Then you will take a busy program and find where its time is going.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every graded screen in this lesson was captured by booting this exact lab and running these exact commands. Your own PIDs and timings will be your own. The lab user is tracer and the machine is named syscall-lab. Your progress in the lab is tracked automatically, so type commands naturally.
The lab has a program called slow_loop already running in the background. It does nothing useful. It sleeps for a second, wakes up, and sleeps again, forever. It stands in for any process that is stuck.
To attach, you need its process id, its PID, the number the kernel uses to name a running program. Then you point strace at that number with -p. Attaching means reading another process, so this one needs sudo. It looks like this:
slow_loop is PID 853
strace: Process 853 attached
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, NULL) = 0The first line is the PID the loop printed when it started. The second is strace telling you it has latched on. That number 853 will be different every time you run this, because the kernel hands out whatever PID is free, so do not memorize it. From the third line on, you are watching the live process.
Attach to the loop and hold still for a moment. The same line prints again and again. That repetition is the whole point: a stuck process is stuck inside one system call, and attaching shows you which one on a loop.
sudo strace -p 853
prompt: tracer@syscall-lab:~$ answer: sudo strace -p 853 ||| strace -p 853 ||| sudo strace -p PID ||| sudo strace -p $(pidof slow_loop) output: clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, NULL) = 0 clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, NULL) = 0 hint: Attach to the running loop by its PID: sudo strace -p 853
Read the line: clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, NULL) = 0. It is one system call, asking the kernel to sleep for one second, tv_sec=1. The = 0 at the end means the sleep finished cleanly. Then the loop asks for the exact same sleep again, which is why the same line keeps scrolling.
This is what blocked looks like from the outside. The process is not frozen and it is not broken. It is sitting inside one call, waiting for the kernel to return, over and over. When a real program hangs, this is the picture you are looking for: the one call it never comes back from.
Press Ctrl-C to detach when you have seen enough. Detaching stops strace, it does not stop the program, so the loop keeps running exactly as it was.
A process is always in one of a few states, and the two that matter here are running and blocked. Running means the kernel is actually giving it the processor. Blocked means it has made a system call that cannot finish yet, so the kernel has parked it and moved on to other work.
slow_loop is blocked. It called clock_nanosleep and asked to sleep for a second, and until that second is up there is nothing for it to do. The kernel sets it aside and wakes it when the time comes. A program waiting on a slow disk, a network reply, or a lock somebody else is holding is blocked in exactly the same way, just inside a different call.
This is the single most useful thing strace -p tells you. When someone says a process is hung, the real question is which call it is blocked in, because that names what it is waiting for. A sleep waits for time. A read on a socket waits for the network. The call is the answer.
There is a wrinkle worth stopping on. The C code for slow_loop calls a function named nanosleep. But the trace you just read said clock_nanosleep. Take a position on why.
>>> It is clock_nanosleep. This is the same libc-versus-kernel gap you have met before: the name in the source is not always the name the kernel sees. The C library takes the old nanosleep request and satisfies it with the newer, more capable clock_nanosleep, and strace reports the kernel's truth. If you picked the first answer, that is the name in the code, but the trace never shows you the code, only the calls that actually cross into the kernel. If you picked the third, the loop is slow because it asks to sleep for a whole second, not because of any doubled-up call; there is exactly one sleep per pass. When you write the answer down later, nanosleep is the name to use, since it names the call family and it is right there inside clock_nanosleep.
The loop was easy: it makes one kind of call and you could see it by eye. A busy program is the opposite. It can make hundreds of thousands of calls too fast to read. For that you do not read the calls at all, you count them, with strace -c. Predict what the count shows you.
>>> One row per kind of call, counted, with the heaviest at the top. The whole run collapses into a short table: this call happened two hundred thousand times, that one happened twice. If you picked the first answer, that is the live trace, and the entire point of -c is to stop reading calls one at a time when there are too many to read. If you picked the third, the summary hides nothing; it counts everything, and it is you who reads the top row to see where the time went. That top row is usually the answer to why is this slow.
The lab has a second program called busy. All it does is ask the kernel for its own process id, with getpid, two hundred thousand times in a row. It is a stand-in for a program burning time in one call. Run strace -c ./busy and a table like this prints when it finishes:
% time seconds usecs/call calls errors syscall
100.00 0.213930 1 200000 getpidThe calls column is the count: getpid was called 200000 times, and that number is the same every run because the program is written to do exactly that. The % time column says that call took all of the program's system-call time. The seconds column, 0.213930 here, is the one number that will not match yours, because it is a live timing and changes every run. Read the counts and the percentages; treat the raw seconds as your machine's own figure.
This is the shape of a real answer to a slow program. One call, getpid, at one hundred percent of the time and two hundred thousand invocations. You did not have to read a single line of the live trace to find it; the summary ranked it for you and put it on top.
In a real investigation the busiest call is your first lead. A program at the top of the table on read is waiting on data. One stuck on futex is fighting over a lock. busy sits on getpid because that is all it was built to do, but the method is the same for any program: run -c, read the top row, and you know where the time lives.
When a process hangs, the natural picture is a program spinning frantically, running as hard as it can, pinning the processor while it fails to get anywhere. It feels stuck because it is working too hard. The trace you read is the proof that this picture is usually wrong.
slow_loop is hung, and it is doing almost nothing. It is parked inside clock_nanosleep, off the processor entirely, using no meaningful CPU at all. The kernel wakes it once a second, it asks to sleep again, and it goes straight back to sleep. A hung process is far more often waiting than working.
That is the shift this lesson turns on. Slow and stuck do not mean busy. A process can sit at zero percent CPU and still be completely wedged, blocked forever on a call that is never going to return. The way you tell the difference is to attach and read the call, which is exactly what you now know how to do.
>>> Almost none. A blocked process is off the processor by definition; the kernel set it aside and is spending those cycles on other work until the sleep is up. If you picked the first answer, that is the busy picture from the last step, and it is the one this lab is built to correct: the loop is asleep, not spinning. If you picked the third, the trace tells you plenty; a process sitting in clock_nanosleep returning = 0 and asking again is the textbook signature of a program waiting, not working. Reading that one call is how you separate a wedged-but-idle process from one that is genuinely burning the machine.
Your lab scores what you can show it found. Two short files hold the two facts from this lesson: the call the stuck process was blocked on, and the busiest call in the profile. Write each name into its file, then read them back.
echo nanosleep > answers/blocked_on.txt
echo getpid > answers/busiest.txtprompt: tracer@syscall-lab:~$ answer: echo nanosleep > answers/blocked_on.txt; echo getpid > answers/busiest.txt; echo 'blocked_on:'; cat answers/blocked_on.txt; echo 'busiest:'; cat answers/busiest.txt ||| cat answers/blocked_on.txt answers/busiest.txt output: blocked_on: nanosleep busiest: getpid hint: Echo each name into its file, then cat both back: echo nanosleep > answers/blocked_on.txt; echo getpid > answers/busiest.txt
Two files, two facts. blocked_on.txt holds nanosleep, the call slow_loop was stuck inside, and busiest.txt holds getpid, the call that ate all of busy's time. The grader reads these, so writing them is not busywork; it is you stating what the attach and the profile told you.
You wrote nanosleep even though the live trace said clock_nanosleep, and both are right: the grader looks for nanosleep, which sits inside the longer name. You recorded the plain name of the call family, which is what a person would say out loud.
One more move rounds out the toolkit. When you attach to a real process in trouble, you often want to keep the trace rather than just watch it scroll past, so you can read it slowly or send it to someone. Add -o to write it to a file, the same flag you used on commands you started:
sudo strace -p 853 -o attached.traceLet it run for a moment, then press Ctrl-C to detach. The file attached.trace now holds the calls the live process made while you were attached, ready to read with less or grep whenever you want. Saving a trace of the attached process is the fifth thing your lab checks.
Stop and notice what changed. You no longer need to start a program to trace it. You can walk up to one that is already running, attach with -p, and read the single call it is blocked in. You can take a busy one, profile it with -c, and name the call eating its time without reading the live trace at all.
That is the exact move a working engineer makes when a service is hung or a job is slow and restarting is not an option. Attach, read the blocking call or the busiest call, and you know what the process is really doing. Everything else is following that thread.
Scaffolding off. No command is printed from here on.
Attach to slow_loop yourself and catch the single call it lives in. It is the same attach you did earlier, the one that needs sudo and the loop's PID, but this time you recall the form and type it without a prompt to copy.
When the call scrolls past, you have your answer for what a stuck process is waiting on.
prompt: tracer@syscall-lab:~$ answer: sudo strace -p 853 ||| strace -p 853 ||| sudo strace -p PID ||| sudo strace -p $(pidof slow_loop) output: clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, NULL) = 0 hint: Attach to the loop by its PID, with sudo: sudo strace -p 853
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, NULL) = 0. The same call, on demand, because you knew where to look. Attaching to a live process and reading the one call it is blocked in is a skill you now own, and it works on any hung program on any machine, not just this loop.
You took a running program you did not start and read what it was doing from the inside. Here is every form this lesson taught:
| Command | What it does |
|---|---|
sudo strace -p 853 | Attach to a running process by PID and print its calls live |
strace -c ./busy | Profile a program, counting each kind of call, busiest on top |
sudo strace -p 853 -o attached.trace | Attach and save the live trace to a file |
And the four facts underneath all of it:
-p.strace -c ranks a run by kind of call, and the top row is your first lead on where the time went.When a process is misbehaving and you cannot restart it, reach for two things first: strace -p PID to see the call it is blocked in, and strace -c on a fresh run to see where the time goes. Between them they answer what is this doing and why is it slow.
You saw clock_nanosleep where the code said nanosleep, and in an earlier lesson you saw openat where the code said open. The next lesson is about that gap directly: why some calls never appear in a trace at all, and the tool that shows the layer strace cannot.
This lesson walked you through the whole loop. You attached to a running process with -p, read the one call it was blocked in, profiled a busy program with -c, named the call eating its time, and saved a trace of the live process to a file. Then you wrote down the two facts the attach and the profile told you.
The lab is that same work on a real machine, with slow_loop already running, busy built and waiting, and an answers directory ready. It scores five things: you attached to a running process, you read the blocking call, you profiled with -c, you recorded the busiest call, and you saved a trace of the attached process.
The objectives name the goal, not the command. You read what needs to be true, recall the form, and type it. Your progress is tracked automatically as you work, so type naturally, and run check-progress whenever you want to see your score.
Launch the lab and read a stuck process from the inside.
Practice Attach and Diagnose in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.