Learn › Linux Foundations › Text Processing

awk

awk - a hands-on Linux lab on a real virtual machine.

Read text as fields with awk: address columns as $1, $2, $0, use the comma in print to space them, and set the divider with -F. Meet the NR and NF counters, filter rows with pattern {action}, and total a column at the END, all in the practice terminal on a tiny staff.txt.

You have spent this module slicing text: cut pulled out columns by number, sort reordered lines, grep matched patterns. Every one of those saw a line as a flat run of characters. This lesson introduces a tool that sees something richer. It looks at a line and sees fields, the separate columns inside it, and lets you name them one by one.

Picture a plain text file of records, where each line is a person and their details sit in columns separated by spaces. You want just the names. Or just the names and cities, skipping the ages. Or only the people over 30. cut struggles the moment the work gets conditional. The tool that does all of this in one short line is awk.

The black boxes in this lesson are a practice terminal: a safe sandbox that only checks the one command each step teaches, so you cannot break anything. Every output shown is exact for one tiny file you will get to know, so what you read here is what the real machine prints.

awk reads a file one line at a time. For each line, it splits the text into fields, its separate columns, using spaces as the divider by default. Then it lets you refer to those fields by position: $1 is the first field, $2 the second, $3 the third, and so on. $0 means the whole line, untouched.

That splitting is the whole idea. Where cut needs you to count character positions or name a delimiter every time, awk hands you ready-made columns you address by number. A field is just one column of one line. $2 is "the second column of whatever line I am looking at right now."

The name awk is not a word. It is the initials of the three people who wrote it at Bell Labs in 1977: Aho, Weinberger, and Kernighan. The same Brian Kernighan of early Unix and C. The tool has counted and reshaped columns of text on Unix systems ever since.

Everything below runs against one small file, staff.txt. It has exactly three lines, each with three space-separated columns: a name, an age, and a city.

alice 30 NYC
bob 25 LA
carol 35 SF

So on the first line, $1 is alice, $2 is 30, and $3 is NYC. The simplest useful awk program prints one field from every line. The instruction goes inside single quotes and curly braces: '{print $1}' means "for each line, print field one." Pull every name out of the file:

awk '{print $1}' staff.txt

prompt: student@linuxcamp:~$ answer: awk '{print $1}' staff.txt output: alice bob carol hint: The program goes in single quotes: awk '{print $1}' staff.txt

Three names, one per line. awk walked the file top to bottom, split each line on its spaces, and printed only column one. The ages and cities never appeared because you never asked for them. The single quotes matter: they hand the whole {print $1} program to awk untouched. Without them the shell would try to make sense of the braces and the $1 itself, and the program would break before awk ever saw it.

You can name several fields in one print. How you join them decides the spacing, and this is the detail that trips people up. Put a comma between two fields and awk prints them with a single space in between. Leave the comma out and it jams them together with nothing between.

You want each person's name and city, dropping the age in the middle. Name field one and field three, with a comma between them:

awk '{print $1, $3}' staff.txt

prompt: student@linuxcamp:~$ answer: awk '{print $1, $3}' staff.txt output: alice NYC bob LA carol SF hint: Two fields, a comma between them: awk '{print $1, $3}' staff.txt

Name, a space, city, for each line. The comma is what put that space there. Write '{print $1 $3}' with no comma and you would get aliceNYC, bobLA, carolSF, all mashed together. That is not a bug; it is awk doing exactly what you said. The comma is the everyday way to keep columns readable, so reach for it whenever you print more than one field.

awk splits on spaces by default, but real data is often separated by something else: a colon, a comma, a tab. The -F flag sets the field separator, the character awk splits on. -F: means "split on colons instead of spaces."

Here is a second tiny file, ports.csv, holding one line with colon-separated columns:

http:80:tcp

With the default space split, awk would see that whole line as one field, because it contains no spaces. Tell awk to split on colons instead, then print field one:

awk -F: '{print $1}' ports.csv

prompt: student@linuxcamp:~$ answer: awk -F: '{print $1}' ports.csv output: http hint: The -F flag sets the separator, then the character: awk -F: '{print $1}' ports.csv

One word, http. By passing -F: you told awk the columns are divided by colons, so $1 became http, $2 would be 80, and $3 would be tcp. Without -F:, awk would split on spaces, find none, and treat http:80:tcp as a single field, so $1 would print the whole line back at you. Whenever your columns are joined by anything other than spaces, -F is the flag that tells awk where one column ends and the next begins.

awk keeps a running tally of two useful numbers while it reads. NR is the number of the current record, that is, the current line number, starting at 1. NF is the number of fields on the current line. Both update automatically as awk moves down the file.

NR lets you pick a line by its position. A program that is just a condition, with no braces, prints the lines that match it. So 'NR==2' means "print only the line whose number is 2." Grab the second line of staff.txt:

awk 'NR==2' staff.txt

prompt: student@linuxcamp:~$ answer: awk 'NR==2' staff.txt output: bob 25 LA hint: NR is the line number, and == tests equality: awk 'NR==2' staff.txt

The whole second line, bob 25 LA. Two things to notice. First, NR==2 is a pattern with no {action} after it. When you give awk a pattern alone, its default action is to print the matching line in full, so you got the entire row, not one field. Second, NR counted for you: line one was skipped, line two matched, line three was skipped. Its partner NF counts fields instead of lines, so on any line of staff.txt, NF is 3, because each line has three columns.

This is where awk outgrows cut. You can put a condition in front of the action, and awk runs the action only on lines that pass the test. The shape is pattern {action}. A field can be part of the test, so you can filter rows by their contents.

You want only the people older than 30. The age sits in field two, so the test is $2 > 30. Leave off the action entirely and awk falls back to its default of printing the whole matching line:

awk '$2 > 30' staff.txt

prompt: student@linuxcamp:~$ answer: awk '$2 > 30' staff.txt output: carol 35 SF hint: The condition compares field two to 30: awk '$2 > 30' staff.txt

One row survives: carol 35 SF, the only person over 30. awk checked field two on each line against 30. Alice at 30 failed (30 is not greater than 30), bob at 25 failed, carol at 35 passed. Because there was no {action}, the default kicked in and the full matching line printed. Flip your mental model here: a bare {action} runs on every line, while a bare pattern prints only the lines that match. Put them together, pattern {action}, and the action runs only on the matches.

awk has two special patterns that are not about matching any line. BEGIN {} runs its action once, before the first line is read. END {} runs its action once, after the last line is read. END is where you print a total you have been building up, line by line.

You want to add up all the ages. The trick is a variable of your own, sum. On every line you add field two to it with sum += $2, which means "add $2 to sum." That runs for each line. Then, once, at the END, you print the total:

awk '{sum += $2} END {print sum}' staff.txt

prompt: student@linuxcamp:~$ answer: awk '{sum += $2} END {print sum}' staff.txt output: 90 hint: Add field two on each line, print at the END: awk '{sum += $2} END {print sum}' staff.txt

One number, 90, the sum of 30, 25, and 35. Watch how the two parts split the work. The plain {sum += $2} block has no pattern, so it ran on every line, quietly growing sum: 30, then 55, then 90. Nothing printed during those lines. Then the END {print sum} block fired exactly once, after the file was done, and printed the final total. That pattern, accumulate on every line and report at the END, is one of the most common things awk is asked to do.

Scaffolding off. No command is printed this time. You have every piece you need.

You are back on staff.txt, the three-line file of name, age, city. You want each person's name and age, and nothing else, no city. Print field one and field two, and keep a space between them so the two columns stay readable.

prompt: student@linuxcamp:~$ answer: awk '{print $1, $2}' staff.txt output: alice 30 bob 25 carol 35 hint: Two fields with a comma between them keeps the space: awk '{print $1, $2}' staff.txt

Name, a space, age, for all three lines. You reached for {print $1, $2} from memory: fields one and two, joined by the comma that puts the space between them. That is the core awk move, naming the exact columns you want and letting the comma space them out, and you typed it with no prompt. It is the same recall the real machine will ask of you.

You earned this cheat sheet. Every row is a form of awk you have already run:

And the vocabulary you can now read on any line: $1 through $NF are the fields, $0 is the whole line, NF is how many fields there are, and NR is the current line number. The one distinction to carry away: a bare pattern prints matching lines, a bare {action} runs on every line, and pattern {action} runs the action only on the matches.

The single quotes around the program are not optional. They keep the shell from touching the $1, the braces, and the spaces inside your awk program. Forget them and the shell eats the braces or blanks out $1 before awk runs, and you get a confusing error or empty output. When an awk command misbehaves, the missing quote is the first thing to check.

The practice terminal has shown you the core of awk: fields addressed by number, the comma that spaces them, -F for a different divider, the NR and NF counters, filtering rows with pattern {action}, and totalling with END. Every one of those you typed yourself.

The Text Processing module ends with one real Linux machine, the text-processing capstone mission, and that is where you reshape columns for real. A VM boots just for you, with its own files. It hands you objectives that use exactly what you practiced across this module: cut, sort, grep, and now awk to split lines into fields and act on them. One difference from here: the mission shows no commands. You read the objective, you recall the command, you type it. That recall is what makes it stick.

Finish the other Text Processing lessons, then go split some real columns for yourself.

Practice awk in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.