Learn › Learn C › C in Practice
gdb - a hands-on Linux lab on a real virtual machine.
Compile with -g and drive a real gdb session on a real crash: run, backtrace, break, info locals, print, continue. Read a SIGSEGV backtrace bottom up, find the NULL pointer that strlen was handed (count_words s=0x0 at bug.c:5), and end the crash with a one-line guard. These are the exact moves you will use when the CSV parser lab segfaults.
Nine lessons ago you compiled your first program. Since then you have handled strings, opened files, and managed heap memory with malloc. Time to meet the moment every C engineer knows well: the program that dies and refuses to say why.
Here is today's suspect. Thirteen lines. The function count_words is supposed to report whether a line of text contains any words. Read it slowly, especially line 5 and line 10.
#include <stdio.h> #include <string.h>
int count_words(char *s) { int len = strlen(s); return (len > 0) ? 1 : 0; }
int main(void) { int n = count_words(NULL); printf("words: %d\n", n); return 0; }
Line 10 hands count_words the value NULL. Remember from the strings lesson: NULL is the pointer that points at nothing, address zero. In real code nobody types NULL there on purpose. It arrives quietly: a fopen that failed, a lookup that found nothing, a malloc the system refused. This lesson pins it to one visible line so you can watch exactly what it does.
The black boxes below are a practice terminal: a safe sandbox that checks the one command each step teaches. Every output is captured from the real lab machine, Debian 12 with gcc 12.2.0 and gdb 13.1. The long hex addresses and process numbers depend on the machine and the run, so yours will differ. Every function name, file name, line number, and variable value will match.
>>> strlen counts bytes by reading memory, starting at the address you give it, until it finds a zero byte. An empty string would be fine: that is a real address holding one zero byte. NULL is different. It is address 0x0, and the very first read touches memory your program does not own. The kernel stops the program on the spot, so the printf on line 11 never runs. Notice the compiler said nothing. Some bugs only exist at run time.
Prove it. Compile and run:
gcc -Wall bug.c -o bug && ./bug
prompt: operator@camp:~$ answer: gcc -Wall bug.c -o bug && ./bug output: Segmentation fault hint: Compile then run in one line: gcc -Wall bug.c -o bug && ./bug
Now ask the shell how the program ended:
prompt: operator@camp:~$ answer: echo $? output: 139 hint: The last exit code lives in $?. Print it with echo $?
Segmentation fault is the kernel speaking, not your program. The program touched memory it does not own, so the kernel killed it with signal 11, named SIGSEGV. The exit code 139 is 128 plus 11: the shell's way of reporting death by signal. One honesty note: the C language itself does not promise this crash. Reading through NULL is undefined behavior, which means the standard allows anything to happen. On Linux, address zero is deliberately kept off limits so this mistake dies loudly instead of quietly corrupting data, and that loud death is what you will debug. Now look at what the message does NOT tell you. No file. No line number. No function name. On thirteen lines you can eyeball the bug. On thirteen thousand you cannot. You need a tool that names the exact line. That tool is gdb.
A debugger is a program that runs your program under supervision. It can pause your program at any moment, and while it is paused, it can read every variable, every function call in progress, every address. gdb, the GNU Debugger, is the standard debugger on Linux.
Think of a flight recorder. After a crash, investigators do not guess. They open the black box and read what every instrument showed at the moment of impact. gdb is the black box for your program, with one upgrade: it can also freeze the flight mid-air and let you walk around the cabin.
gdb needs a map to be useful. Compiled machine code has no variable names and no line numbers; the compiler threw those away. The gcc flag -g packs the map back in. It stores debug information inside the binary: which instruction came from which line of which file, and where each variable lives.
From today, compile with gcc -Wall -g. The -g flag does not slow your program down; it only makes the file on disk bigger. Without it, gdb still runs your program, but the argument values and the file and line numbers are gone from its answers. You will prove that for yourself in a few minutes.
You already debug. You add a printf, recompile, run, read, delete the printf. That works, and working engineers do it every day. It has three limits. A crash can kill the program before your message ever appears. Every guess costs a recompile. And you must know in advance which variable to look at.
gdb removes the guessing. You reach for it in three situations. A crash: gdb names the exact file, line, and function, plus the value that caused it. A wrong answer: pause the program mid-run and read the variables to see where the numbers went bad. Unfamiliar code: pause inside a function and see who called it, with what arguments.
Later, in the system programming track, you will point this same tool at core dumps and live processes. The commands you learn today are the same ones used there.
First, rebuild bug.c with the debug map inside:
gcc -Wall -g bug.c -o bug
prompt: operator@camp:~$ answer: gcc -Wall -g bug.c -o bug output: hint: The same compile line plus the -g flag: gcc -Wall -g bug.c -o bug
There are two ways to drive gdb. Interactive: gdb ./bug opens a (gdb) prompt where you type commands one at a time and quit with q. Batch: gdb --batch runs a list of commands and exits by itself. Each -ex flag carries one gdb command, executed in order, left to right. Same commands either way. The practice terminal takes one line, so this lesson uses batch mode; on the real VM you can use either.
You need two gdb commands. run starts the program inside gdb. bt, short for backtrace, prints the chain of function calls active at this exact moment.
>>> Frame #0 is where the CPU actually was at the moment of death: inside __strlen_avx2, the C library's hand-tuned version of strlen. The wrong reflex is to read that top line and conclude the C library is broken. It is not. It did exactly what strlen does, read memory starting at the address it was given, and the poisoned address came from your code. Your bug lives one frame below it, in count_words. This is the single most useful backtrace skill: the top frame is often library code, and your bug is the first frame below it that names a file you wrote.
Run it and get the backtrace, one line:
gdb --batch -ex run -ex bt ./bug
prompt: operator@camp:~$ answer: gdb --batch -ex run -ex bt ./bug output: [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Program received signal SIGSEGV, Segmentation fault. __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:76 76 ../sysdeps/x86_64/multiarch/strlen-avx2.S: No such file or directory. #0 __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:76 #1 0x0000555555555161 in count_words (s=0x0) at bug.c:5 #2 0x0000555555555182 in main () at bug.c:10 hint: Batch mode with two commands: gdb --batch -ex run -ex bt ./bug
Walk it line by line. The two libthread_db lines are gdb loading thread support: noise, ignore them. Program received signal SIGSEGV is the same crash as before, but with one huge difference: gdb caught it. The program is not gone; it is frozen at the exact faulting instruction. The No such file or directory line looks alarming but is not an error in your program. gdb wanted to display line 76 of the C library's own source file. That source is not installed on this machine, so gdb says so. Ignore it too. Frame #0 names __strlen_avx2 in a .S file, which is assembly source: the C library's fast strlen. Not where your bug lives, just where the pain surfaced. Frame #1 is the treasure: count_words (s=0x0) at bug.c:5. Your file, your line, and the argument it received: s=0x0, address zero, NULL. Frame #2 names the caller: main () at bug.c:10. One command took you from a silent crash to the exact line and the exact poisoned value.
The long hex numbers like 0x0000555555555161 are per-machine and per-build, so yours may differ. gdb turns off address randomization by default, which is why they stay stable between runs on one machine. Everything else, the function names, s=0x0, bug.c:5, bug.c:10, is what you should see too.
A backtrace is the list of function calls in progress right now, newest first. Each entry is called a frame. Remember from the functions and the stack lesson: every call pushes a frame holding that function's arguments and locals, and every return pops it. bt simply prints the frames that are live at the frozen moment.
To reconstruct the story, read it bottom up. Frame #2: main was running and reached line 10. Frame #1: line 10 called count_words, which reached line 5. Frame #0: line 5 called strlen, and inside strlen the CPU hit unowned memory. main called count_words, count_words called strlen. The whole plot, three lines.
To find the bug, start at #0 and move down until the first frame that names a file you wrote. Here that is #1: count_words (s=0x0) at bug.c:5. Library frames above your first frame are almost never the bug. The values in the parentheses are the arguments the function received, and s=0x0 is the smoking gun.
{ "kind": "stack", "frames": [ { "fn": "#0 __strlen_avx2 (C library)", "locals": [ { "name": "reading at", "value": "0x0" } ], "ret": "0x0000555555555161 (back into count_words)" }, { "fn": "#1 count_words, bug.c:5", "locals": [ { "name": "s", "value": "0x0 (NULL)" } ], "ret": "0x0000555555555182 (back into main)" }, { "fn": "#2 main, bug.c:10", "locals": [ { "name": "n", "value": "never assigned" } ] } ] }
One more detail earns its keep: the hex numbers printed on frames #1 and #2 are return addresses, the exact spot where execution resumes when the frame above returns. That is how gdb rebuilds the chain. Each frame remembers where to go back to.
One shortcut will tempt you for years, so kill it now. gdb seems to run any binary, so why bother rebuilding with -g? Your very first compile in this lesson, gcc -Wall bug.c -o bug, had no -g in it. Suppose you had pointed gdb straight at that binary. gdb will not refuse. Predict what it costs you.
>>> gdb never refuses. It happily runs any binary, which is exactly why this shortcut tempts people. And the crash really is the same: same signal, same faulting instruction. What changes is what gdb can TELL you about it. Function names survive without -g because the linker keeps a small name table for its own use. But the map from instructions to files and lines, and the notes on where each variable lives, exist only if -g wrote them in. No -g, no s=0x0, no bug.c:5.
See it for yourself. Build the no-map version and ask for the same backtrace, one line:
gcc -Wall bug.c -o nomap && gdb --batch -ex run -ex bt ./nomap
prompt: operator@camp:~$ answer: gcc -Wall bug.c -o nomap && gdb --batch -ex run -ex bt ./nomap output: [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Program received signal SIGSEGV, Segmentation fault. __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:76 76 ../sysdeps/x86_64/multiarch/strlen-avx2.S: No such file or directory. #0 __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:76 #1 0x0000555555555161 in count_words () #2 0x0000555555555182 in main () hint: Compile to nomap without -g, then the same batch line: gcc -Wall bug.c -o nomap && gdb --batch -ex run -ex bt ./nomap
Put this next to the backtrace you got with -g. Same signal. Same __strlen_avx2 frame. Look closely at the hex numbers: the SAME return addresses, ending in 161 and 182. The -g flag did not move or change a single instruction; this is the same crashing logic byte for byte, minus the map. And without the map, frame #1 collapses to count_words (): no poisoned argument, no file, no line. On thirteen lines you could still find the bug by staring. On a real codebase, count_words () alone means hours of guessing that count_words (s=0x0) at bug.c:5 would have skipped. That is the whole case in one sentence: gdb can only answer in your files, lines, and values if -g packed that map into the binary, so gcc -Wall -g is the first move of every debugging session, not an optional extra.
gdb is not only for crashes. Its second superpower is pausing a program that is working fine, so you can inspect it mid-flight. The pause marker is called a breakpoint: you attach it to a function or a line, and when execution reaches that spot, gdb freezes the program there.
The test subject is frames.c, the same tiny program from the functions and the stack lesson:
#include <stdio.h> int add(int a, int b) { int sum = a + b; return sum; } int main(void) { printf("%d\n", add(2, 40)); return 0; }
Four commands work as a team. break add sets the trap on the function add. run starts the program, which stops the instant add is entered. info locals lists every local variable in the current frame with its current value. continue releases the pause and lets the program finish. There is also print for one thing at a time: print a, print sum, or a whole expression like print a + b.
>>> A breakpoint pauses BEFORE the marked line runs. The variable sum already has its slot in the stack frame, but nothing has been stored there yet, so info locals shows whatever that slot happened to hold. On this run it held 0. Do not lean on that: C makes no promise about an unassigned local. It could hold any value. Trust a local only after its assignment line has executed.
Compile frames.c with gcc -Wall -g frames.c -o frames, then run the whole session in one line:
gdb --batch -ex 'break add' -ex run -ex bt -ex 'info locals' -ex continue ./frames
prompt: operator@camp:~$ answer: gdb --batch -ex 'break add' -ex run -ex bt -ex 'info locals' -ex continue ./frames output: Breakpoint 1 at 0x1143: file frames.c, line 2. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Breakpoint 1, add (a=2, b=40) at frames.c:2 2 int add(int a, int b) { int sum = a + b; return sum; } #0 add (a=2, b=40) at frames.c:2 #1 0x0000555555555166 in main () at frames.c:3 sum = 0 42 [Inferior 1 (process 961) exited normally] hint: Five -ex commands in order: gdb --batch -ex 'break add' -ex run -ex bt -ex 'info locals' -ex continue ./frames
Top to bottom. Breakpoint 1 at 0x1143 confirms the trap is planted inside add, at the start of its body. gdb deliberately skips past the few setup instructions at a function's entrance, so the arguments are already in place when it stops. The libthread_db pair is the same loading noise as before. run starts the program, and Breakpoint 1, add (a=2, b=40) at frames.c:2 is the frozen moment: gdb hands you the arguments for free, a is 2, b is 40. The next line echoes the source line about to run. bt shows two frames: add on top, main below it at frames.c:3, exactly the crash-reading skill from the last step, now on a healthy program. info locals reports sum = 0, the leftover slot from the prediction. continue releases the pause: the program computes, prints 42, and exits. The final line is gdb confirming a clean exit; inferior is its word for the program under supervision, and the process number changes every run. Nothing crashed. You paused a working program, read its mind, and let it go.
Back to bug.c. The backtrace said everything: count_words trusted its input, and its input was NULL. The professional fix is a guard: check the pointer before touching it. Here is the file with one added line, line 5:
#include <stdio.h> #include <string.h>
int count_words(char *s) { if (s == NULL) return 0; int len = strlen(s); return (len > 0) ? 1 : 0; }
int main(void) { int n = count_words(NULL); printf("words: %d\n", n); return 0; }
if (s == NULL) return 0; says: no text means no words. The poisoned value never reaches strlen. Rebuild and run:
gcc -Wall -g bug.c -o bug && ./bug
prompt: operator@camp:~$ answer: gcc -Wall -g bug.c -o bug && ./bug output: words: 0 hint: The same build-and-run line as before: gcc -Wall -g bug.c -o bug && ./bug
No crash. An answer: words: 0. Two lessons live in that one line. First, the pattern: functions that receive pointers guard them at the top, and production C is full of exactly this check. Second, the guard treats the symptom. Frame #2 of the backtrace also told you WHERE the NULL came from: main, line 10. In real work you follow that lead too, and ask why the caller had nothing to pass. gdb gives you both ends of the bug.
Three messages will follow you through your whole C career. Learn to read them calmly.
Segmentation fault and an exit code of 139.
What it means: the kernel killed your program with signal 11, SIGSEGV, because it touched memory it does not own. 139 is 128 plus 11.
What to check first: recompile with -g, run gdb --batch -ex run -ex bt ./program, and read down the backtrace to the first frame in your own file.
Program received signal SIGSEGV, Segmentation fault.
What it means: the same crash, caught inside gdb. The program is not gone; it is frozen at the faulting instruction, waiting for your questions.
What to check first: the backtrace. In batch mode your -ex bt already printed it. In an interactive session, type bt, then info locals and print on anything suspicious.
__strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:76
What it means: the crash surfaced inside the C library's assembly code (a .S file is assembly source). The library is not broken. It received a poisoned argument f
Practice Debugging with gdb in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.