Learn › Linux Foundations › Text Processing
Part 1 of 3: printf and its format slots, grep with anchors, the dot, character classes and -E, then -i, -v and -n, cut by field and by character, and sort with -n and -r (Sections 13.1 to 13.12). No machine test here; it completes on the final step. The chapter lab waits at the end of Part 3.
A service log has twelve order events. If you need the warning lines in time order, start by reading the file, select the lines that matter, and sort the selected text. This part teaches the first tools in that chain.
Sections in this part: 13.1–13.2 printf; 13.3–13.9 grep and its patterns; 13.10–13.11 cut; 13.12 sort. Parts 2 and 3 continue the same chapter with counting, combining, comparing and changing text.
This is Part 1 of 3. It covers Sections 13.1 to 13.12. Those sections teach the tool that prints text in a shape you choose and the tool that finds lines. They also teach the two tools that take a file apart by column and put it back in order. Part 2 covers Sections 13.13 to 13.18, which count, translate and combine whole files. Part 3 covers Sections 13.19 to 13.23 and ends in the chapter lab, which covers all twenty-three sections.
What you already know: Chapter 5 taught you to read a file with cat, head and tail. Chapter 11 taught you the pipe, which sends the output of one command into the next, and the operators > and <, which connect a command to a file. Some tools read a named file or standard input and write lines to standard output. printf instead creates text from its arguments, and patch changes a file. When one tool writes to standard output and the next reads standard input, a pipe joins them; > records the final result in a file.
Objectives
After this part you can:
Use Start machine to open the real practice server. The recorded output beside each command is an example from the prepared fixture; run the command in Terminal and inspect what your machine actually reports. Commands can change its files, so check the result before continuing. A reconnect may start a new shell. Run pwd to check where you are and cd ~ to return home. Prompt colors and symbols can differ from the examples. Paths beginning ~/analytics/ work from any directory; shorter analytics/ alternatives assume you are in /home/admin. This part ends on its final step. The chapter assessment follows Part 3 on a separate fresh machine.
Five questions before the material. A wrong answer costs nothing: the explanation states the fact and names the section that teaches it.
>>> The pipe from Chapter 11 passes output bytes to the next command's input. That works when the receiving command reads standard input. printf creates output from arguments, while grep and sort can read a pipe. Section 13.14 builds a three-stage chain; the tools do not all take input in the same way.
>>> grep works line by line. If the pattern is found anywhere in a line, the WHOLE line is printed, and if it is not, nothing is printed for that line. Section 13.3.
>>> A pattern can carry characters that mean a position rather than a letter. The caret means the start of the line and the dollar means the end. They are called anchors. Section 13.4.
>>> cut splits every line on a delimiter you name and prints the fields you ask for, by number. Section 13.10.
>>> sort compares text, not numbers, unless it is told otherwise. Character by character, 103 comes before 12 because the second character 0 is before 2, and both come before 7. The option -n makes sort read each line as a number instead. Section 13.12.
A dataset is a file of records that a program is expected to read. The chapter works on the reporting area of server1, the directory ~/analytics, which holds one small dataset for each tool it teaches. Nothing in it is large: every file is short enough to print on one screen, so you can always check the answer by eye.
Check pwd; run cd ~ if needed. Then list ~/analytics:
prompt: admin@server1:~$ answer: ls analytics ||| ls ~/analytics ||| ls analytics/ output: README.txt config orders.log report tickets.txt bakers-arrival.txt counts.txt prices.txt reset times.txt bakers.txt notes.txt products.csv sold-tuesday.txt codes.txt notice.txt readings.txt sold-wednesday.txt hint: Section 2.2. The listing tool from Chapter 2, then the reporting directory ~/analytics (or analytics from your home directory).
Eighteen entries. config and report are directories; the rest are files. report is empty and is where finished results go. reset is a small program that puts the area back the way it started, which matters once you begin changing files. Your terminal may lay the names out in a different number of columns; the names and their order are the same.
The README of the directory says what each file holds. Read ~/analytics/README.txt now: every exercise in the chapter uses one of these files, and knowing what is in them is most of the work.
prompt: admin@server1:~$ answer: cat analytics/README.txt ||| cat ~/analytics/README.txt output: analytics: the reporting area of server1 ======================================== The web server writes logs and the shop exports spreadsheets. This directory holds the small datasets Chapter 13 processes, one per tool.
orders.log one line per order event: time, account, level, text products.csv the product list, comma delimited, with a header row tickets.txt one product name per order taken, in the order taken readings.txt oven temperatures recorded overnight, one per line notes.txt a shift note that arrived in lower case codes.txt crate codes with a dash between every character times.txt five event times, and counts.txt the five events, in the same order prices.txt product and price, sorted bakers.txt product and baker, sorted bakers-arrival.txt the same six lines in the order they arrived sold-tuesday.txt products sold on Tuesday, sorted sold-wednesday.txt products sold on Wednesday, sorted notice.txt a shop notice naming one product line config/ two versions of the site configuration, the copy that gets patched, and the patch itself report/ where finished results are filed. Empty to start with
Run analytics/reset at any time to put report/, config/site.conf and notice.txt back the way they started. hint: Section 5.1. The printing tool from Chapter 5, then the path ~/analytics/README.txt.
Every file named there is used by an exercise in one of the three parts. The last paragraph is worth remembering: ~/analytics/reset empties report, puts back the unpatched configuration and puts back the notice, so no mistake in this chapter is permanent.
printf prints text according to a format, which is the first argument you give it. The format is a piece of text with slots in it. A slot begins with a percent sign, and printf fills each slot with one of the arguments that follow the format.
The slot %s means a string: put the next argument here, as it is. The two characters \n in a format mean a newline.
printf prints exactly what the format says and nothing else. That is the difference between it and echo, which adds a newline at the end unless you tell it not to.
prompt: admin@server1:~$ answer: printf '%s\n' 'NORTHWIND ORDER DIGEST' output: NORTHWIND ORDER DIGEST hint: Section 13.1. The formatting tool, then the format '%s\n' in single quotes, then 'NORTHWIND ORDER DIGEST' in quotes as one argument.
One line. The format %s\n says: one string, then a newline. The single quotes around the format stop the shell from touching the backslash, and the single quotes around the text keep the three words as one argument (Section 8.17 covers quoting).
Leave the \n out and printf prints no newline. The next thing on your screen, which is your prompt, starts on the same line. This is not a fault: it is what lets printf build a line in pieces.
prompt: admin@server1:~$ answer: printf 'no newline at the end' output: no newline at the end hint: Section 13.1. The same tool with the plain format 'no newline at the end': no slot and no \n.
On the real server your next prompt appears immediately after the final d, on the same line. Nothing is broken. Press Enter and the prompt moves down.
If you give printf more arguments than the format has slots, it uses the format again, from the start, until the arguments run out. One format with one slot and three arguments prints three lines.
prompt: admin@server1:~$ answer: printf '%s\n' BAGUETTE CIABATTA RYE output: BAGUETTE CIABATTA RYE hint: Section 13.1. The format '%s\n' of Exercise 13-3, then the three names BAGUETTE CIABATTA RYE as three arguments.
Three lines from one format. This behavior is what makes printf useful in a loop or with the output of another command: you write the shape once and feed it as many values as you have.
A format can hold more than one slot, and it can hold ordinary text between the slots. printf takes the arguments in order, one per slot.
prompt: admin@server1:~$ answer: printf '%s costs %s\n' BAGUETTE 3.20 output: BAGUETTE costs 3.20 hint: Section 13.2. The format '%s costs %s\n', then the two values BAGUETTE and 3.20.
The words costs and the space around it are ordinary text in the format and are printed as they are. The two %s slots took the two arguments in the order they were written.
A number written between the percent sign and the letter is the width: the smallest number of characters the slot takes. A shorter value is padded with spaces on the left. A minus sign in front of the width pads on the right instead, which lines the next column up.
prompt: admin@server1:~$ answer: printf '%-10s %s\n' 0412 'crate signed in' output: 0412 crate signed in hint: Section 13.2. The format '%-10s %s\n', then 0412 and 'crate signed in' in quotes.
0412 is four characters, so printf added six spaces to reach ten, then printed the space in the format and the second value. Run it for several rows and the second column starts in the same place on every line. Section 13.21 shows a tool that does this for a whole file without a format.
The slot %d means a decimal number. It accepts a width the same way, and a 0 in front of the width pads with zeros instead of spaces.
prompt: admin@server1:~$ answer: printf '%04d\n' 42 output: 0042 hint: Section 13.2. The format '%04d\n', then the number 42.
Two zeros were added to make four digits. This is how identifiers and times are kept the same width so they sort correctly, which matters in Section 13.12.
printf writes to stdout like any other command. A stream is a flow of text a program reads from or writes to, and stdout is the stream a program sends its normal output to (Section 11.1). The > operator from Section 11.2 attaches that stream to a file instead of to the screen.
The first result of the chapter goes in ~/analytics/report, which is where every finished result belongs.
prompt: admin@server1:~$ answer: printf '%s\n' 'NORTHWIND ORDER DIGEST' > analytics/report/header.txt ||| printf '%s\n' 'NORTHWIND ORDER DIGEST' > ~/analytics/report/header.txt output: hint: Section 13.2. Exercise 13-3 again, then the redirect > of Section 11.2 and the path ~/analytics/report/header.txt.
Nothing printed, because stdout went to the file. This is the first of the twenty-five objectives of the chapter lab.
Check what landed in ~/analytics/report/header.txt with cat (Section 5.1). Reading the file back is the only proof that the right bytes landed, and it costs one command.
prompt: admin@server1:~$ answer: cat analytics/report/header.txt ||| cat ~/analytics/report/header.txt output: NORTHWIND ORDER DIGEST hint: Section 5.1. The printing tool and the file you just wrote, ~/analytics/report/header.txt.
One line, with the newline the format asked for. Without the \n the file would hold twenty-two characters and no line ending, and most tools would treat it as an incomplete line.
grep prints the lines of a file that contain a pattern. A pattern is a description of text. The simplest pattern is a plain word, which describes itself.
grep reads one line at a time. A line that contains the pattern anywhere is printed WHOLE.
The file it works on here is ~/analytics/orders.log, one line per order event. Each line holds a time, an account, a level and then free text. Read it first, because every pattern in the next six sections is aimed at these twelve lines.
prompt: admin@server1:~$ answer: cat analytics/orders.log ||| cat ~/analytics/orders.log output: 0412 wholesale ROUTINE crate code AX-7 signed in 0418 storefront ROUTINE delivery slip printed 0425 wholesale ROUTINE pallet count confirmed 0433 catering PRIORITY oven two is below temperature 0447 storefront ROUTINE till float counted 0502 wholesale ROUTINE courier arrival confirmed 0515 wholesale PRIORITY short delivery on lot 4472 0528 storefront ROUTINE flour ledger balanced 0544 catering PRIORITY van without refrigeration on lot 8 confirmed 0551 wholesale ROUTINE crate code AR-7 verified 0600 STOREFRONT NOTICE a customer asked about the closed door 0612 catering PRIORITY oven two recovered on lot 4472 confirmed hint: Section 5.1. The printing tool and the log ~/analytics/orders.log.
Twelve lines. The first field is a time written as four digits. The second is the account that raised the event. The third is one of three levels: ROUTINE, PRIORITY or NOTICE. The rest of the line is a sentence.
Line 11 writes the account STOREFRONT in capitals. Section 13.7 is about that difference.
Give grep a pattern and a file name, in that order. The pattern comes first because grep can be given many files. The file for this section and the next six is ~/analytics/orders.log, and the pattern is PRIORITY.
prompt: admin@server1:~$ answer: grep PRIORITY analytics/orders.log ||| grep PRIORITY ~/analytics/orders.log output: 0433 catering PRIORITY oven two is below temperature 0515 wholesale PRIORITY short delivery on lot 4472 0544 catering PRIORITY van without refrigeration on lot 8 confirmed 0612 catering PRIORITY oven two recovered on lot 4472 confirmed hint: Section 13.3. The line-finding tool, the pattern PRIORITY, then ~/analytics/orders.log. Pattern first, file second.
Four lines out of twelve, printed whole. grep did not print the word PRIORITY; it printed every line the word appears in, in the order the file holds them. These four results are whole lines in file order. grep PRIORITY searches the entire line, so this output alone does not prove PRIORITY is the level field; Section 13.23 will test that field directly with awk.
A pattern that is in no line produces no output. grep says nothing at all: no message, no blank line. Silence is the answer. Search ~/analytics/orders.log for SOURDOUGH, a product name this log does not hold.
prompt: admin@server1:~$ answer: grep SOURDOUGH analytics/orders.log ||| grep SOURDOUGH ~/analytics/orders.log output: hint: Section 13.3. The shape of Exercise 13-12 with the pattern SOURDOUGH against ~/analytics/orders.log.
Nothing. SOURDOUGH is a product name and appears in other files of the area, but not in this log.
grep also sets its exit code to 1 when it finds nothing. An exit code is the number a command leaves behind to say whether it worked (Chapter 10), and that is what makes grep useful in an && or || chain.
A pattern can carry characters that stand for a POSITION instead of a letter. These are called anchors. The caret ^ means the start of the line, so ^05 matches only where 05 is the first two characters. Without it, a line holding 05 anywhere would match.
Anchors are why patterns are quoted. The shell has its own meaning for several of these characters, and single quotes hand the pattern to grep untouched. The file is ~/analytics/orders.log, as it is for every search up to Section 13.9.
prompt: admin@server1:~$ answer: grep '^05' analytics/orders.log ||| grep '^05' ~/analytics/orders.log output: 0502 wholesale ROUTINE courier arrival confirmed 0515 wholesale PRIORITY short delivery on lot 4472 0528 storefront ROUTINE flour ledger balanced 0544 catering PRIORITY van without refrigeration on lot 8 confirmed 0551 wholesale ROUTINE crate code AR-7
Practice Chapter 13, Part 1. Text Processing: printf, grep, cut and sort in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.