Learn › Linux Foundations › Text Processing
Part 2 of 3: uniq and the law that its input must be sorted, uniq -c with sort -nr for the count and rank chain, tr for translating and deleting characters, and paste, join and comm for combining two files (Sections 13.13 to 13.18). No machine test here; it completes on the final step. The chapter lab waits at the end of Part 3.
Suppose a report repeats an account name on several lines. Before you count accounts, put matching names next to each other; otherwise a line-by-line duplicate check misses repeats. This part builds that count and then combines two prepared files.
Sections in this part: 13.13–13.14 uniq; 13.15 tr; 13.16 paste; 13.17 join; 13.18 comm. Part 1 taught selecting and sorting lines; Part 3 compares and changes files.
This is Part 2 of 3. Part 1 covered Sections 13.1 to 13.12, the tools that work on one file and print a part of it. This part covers Sections 13.13 to 13.18, which count what is in a file, change it character by character, and combine TWO files into one. Part 3 covers Sections 13.19 to 13.23 and ends in the chapter lab, which covers all twenty-three sections.
What you already know: Section 13.12 taught sort, which prepares input for uniq, join and comm. tr and paste do not need sorted input. Section 13.10 taught fields and delimiters; Chapter 11 taught pipes and redirects. Use each connection only where the command needs it.
Objectives
After this part you can:
uniq needs adjacent repeats, and use sorting when repeats are scattered.Use Start machine to open the real practice server. The recorded output beside each command is an example from the prepared fixture; run the command in Terminal and inspect what your machine actually reports. Commands can change its files, so check the result before continuing. A reconnect may start a new shell. Run pwd to check where you are and cd ~ to return home. Prompt colors and symbols can differ from the examples. Paths beginning ~/analytics/ work from any directory; shorter analytics/ alternatives assume you are in /home/admin. This part ends on its final step. The chapter assessment follows Part 3 on a separate fresh machine.
Five questions before the material. A wrong answer costs nothing: the explanation states the fact and names the section that teaches it.
>>> uniq compares a line only with the line immediately before it. Adjacent repeats are enough; the file does not strictly have to be sorted. Sorting is the reliable preparation when duplicates are scattered. Section 13.13.
>>> Three small tools in a row answer it: sort groups the identical lines, uniq -c replaces each group with a count, and sort -nr puts the largest count first. Section 13.14.
>>> tr reads standard input and writes standard output, and takes no file name at all. The < operator of Section 11.5 attaches a file to standard input, and a pipe does the same job from another command. Section 13.15.
>>> join reads two files, matches the lines whose first field is the same, and prints that field once followed by the rest of both lines. It is the operation a database calls a join, done on plain text. Section 13.17.
>>> comm walks both files at once, comparing the line it is holding from each. That only works if both are already in the same order, and comm says so out loud when they are not. Section 13.18.
uniq removes repeated lines. It compares each line with the one immediately before it, and if they are the same it prints only the first. It never looks any further back than one line.
The ticket list from Section 13.12 has fifteen lines and five different names. Two BAGUETTE lines happen to sit next to each other, at lines 2 and 3, and the rest of the repeats are scattered. Check pwd; run cd ~ if needed. Run uniq on ~/analytics/tickets.txt as it is:
prompt: admin@server1:~$ answer: uniq analytics/tickets.txt ||| uniq ~/analytics/tickets.txt output: CIABATTA BAGUETTE RYE SOURDOUGH CIABATTA BAGUETTE FOCACCIA RYE BAGUETTE SOURDOUGH CIABATTA BAGUETTE RYE CIABATTA hint: Section 13.13. The repeat-collapsing tool and ~/analytics/tickets.txt, with no sorting in front of it.
Fourteen lines, not five. Exactly one line was removed: the second of the two BAGUETTE lines that were next to each other. Every other repeat is still there, because uniq never saw it as a repeat. uniq compares a line only with the line before it, and that one-line window is the whole of what it does.
Sorting puts every repeat next to its twin, which is exactly the state uniq needs. The pipe of Section 11.10 hands the sorted lines straight to uniq without a file in between. Sort ~/analytics/tickets.txt and pipe it into uniq.
prompt: admin@server1:~$ answer: sort analytics/tickets.txt | uniq ||| sort ~/analytics/tickets.txt | uniq output: BAGUETTE CIABATTA FOCACCIA RYE SOURDOUGH hint: Section 13.13. The ordering tool and ~/analytics/tickets.txt, the pipe | of Section 11.10, then the repeat-collapsing tool with no file name.
Five lines, one per product. uniq was given no file name here: it read standard input, which the pipe filled with the output of sort. sort file | uniq is the usual way to reduce a list to its distinct lines.
sort can do both jobs itself. The option -u sorts and keeps only the first of each group of identical lines. Run it on ~/analytics/tickets.txt.
prompt: admin@server1:~$ answer: sort -u analytics/tickets.txt ||| sort -u ~/analytics/tickets.txt output: BAGUETTE CIABATTA FOCACCIA RYE SOURDOUGH hint: Section 13.13. The ordering tool with the option -u, which keeps one of each group, then ~/analytics/tickets.txt.
The same five lines. sort -u is shorter, and it does the work in one command instead of two. The two-command form is worth knowing anyway, because uniq has options that sort -u has not, and the next section uses one of them.
The redirect goes at the end of the whole pipeline, after the last command in it. File the list of names in ~/analytics/report/tickets-unique.txt.
prompt: admin@server1:~$ answer: sort analytics/tickets.txt | uniq > analytics/report/tickets-unique.txt ||| sort ~/analytics/tickets.txt | uniq > ~/analytics/report/tickets-unique.txt output: hint: Section 13.13. Exercise 13-39, then > and the path ~/analytics/report/tickets-unique.txt.
Silent. The redirect belongs to the LAST command of the pipeline, which is uniq, so it is uniq output that lands in the file (Section 11.13). Read it back and it holds the five product names.
The option -d turns uniq around: it prints only the lines that DO repeat, one line for each such group, and says nothing about the lines that appear once. Run it on the sorted ~/analytics/tickets.txt.
prompt: admin@server1:~$ answer: sort analytics/tickets.txt | uniq -d ||| sort ~/analytics/tickets.txt | uniq -d output: BAGUETTE CIABATTA RYE SOURDOUGH hint: Section 13.13. Exercise 13-39 with the option -d on uniq, which prints only the lines that repeat.
Four names. FOCACCIA is missing because it was ordered once. -d is what you reach for when you are hunting duplicates in a list that should have none, such as user names or identifiers.
The option -c counts. Instead of removing a run of identical lines, uniq replaces it with one line carrying the count of that run in front of it, padded on the left so the numbers line up. Count how often each product appears in ~/analytics/tickets.txt.
prompt: admin@server1:~$ answer: sort analytics/tickets.txt | uniq -c ||| sort ~/analytics/tickets.txt | uniq -c output: 5 BAGUETTE 4 CIABATTA 1 FOCACCIA 3 RYE 2 SOURDOUGH hint: Section 13.14. Exercise 13-39 with the option -c on uniq, which puts the size of each run in front of the line.
Five lines, each one a count and a name. They are in alphabetical order, because that is the order sort put them in and uniq never reorders anything. Fifteen tickets, and 5 plus 4 plus 1 plus 3 plus 2 is fifteen.
The interesting order is by count, largest first, and Section 13.12 has the tool for it. sort -nr reads the start of each line as a number and reverses the result, and the count is at the start of each line, which is why uniq -c puts it there. Rank the products of ~/analytics/tickets.txt.
prompt: admin@server1:~$ answer: sort analytics/tickets.txt | uniq -c | sort -nr ||| sort ~/analytics/tickets.txt | uniq -c | sort -nr output: 5 BAGUETTE 4 CIABATTA 3 RYE 2 SOURDOUGH 1 FOCACCIA hint: Section 13.14. Exercise 13-43, a second pipe, then the ordering tool with the numeric option and the reversing option written together as -nr.
The same five counts, now in count order: BAGUETTE at 5, then CIABATTA, RYE, SOURDOUGH, and FOCACCIA at 1.
The chain sort | uniq -c | sort -nr answers the question "which one happened most often" for any list of lines. The three stages are always the same: sort groups the identical lines, uniq -c counts each group, and sort -nr ranks the counts.
tr works on characters, not on lines. Given two sets, it replaces every character of the first set with the character in the same position of the second set. It is the tool for changing the case of text, and for throwing away characters you do not want.
tr takes NO file name. It reads standard input and writes standard output, and nothing else, so a file reaches it through < or through a pipe. The file here is ~/analytics/notes.txt, a shift note that arrived in lower case.
prompt: admin@server1:~$ answer: cat analytics/notes.txt ||| cat ~/analytics/notes.txt output: priority note from the wholesale desk oven two ran below temperature at 0433 crate codes ax-7 and ar-7 are in the yard hold the morning bake until the oven is signed off hint: Section 5.1. The printing tool and the shift note ~/analytics/notes.txt.
Four lines, all in small letters, with digits and punctuation among them.
The set a-z is a range: the twenty-six small letters. The set A-Z is the twenty-six capitals. Character one of the first set becomes character one of the second, and so on down both sets. Print ~/analytics/notes.txt in capitals.
prompt: admin@server1:~$ answer: tr 'a-z' 'A-Z' < analytics/notes.txt ||| tr 'a-z' 'A-Z' < ~/analytics/notes.txt output: PRIORITY NOTE FROM THE WHOLESALE DESK OVEN TWO RAN BELOW TEMPERATURE AT 0433 CRATE CODES AX-7 AND AR-7 ARE IN THE YARD HOLD THE MORNING BAKE UNTIL THE OVEN IS SIGNED OFF hint: Section 13.15. The character tool, the set a-z, the set A-Z, then the input redirect < of Section 11.5 with ~/analytics/notes.txt, because this tool takes no file name.
Every letter is now a capital. The digits 0433 and the dashes in ax-7 came through untouched, because they are not in the first set and tr leaves anything it was not told about alone. The FILE is unchanged: tr wrote to your screen.
The option -d deletes every character of the set instead of translating it. Only one set is given. The file for this is ~/analytics/codes.txt, a list of crate codes that arrived with a dash between every character.
prompt: admin@server1:~$ answer: cat analytics/codes.txt ||| cat ~/analytics/codes.txt output: A-X-7-7-4-2 C-I-A-4-4-1 R-Y-E-0-9-3 hint: Section 5.1. The printing tool and the crate codes ~/analytics/codes.txt.
Three codes of six characters each, with five dashes in every line.
One set is enough for -d: the characters to delete. Here that set is a single dash, and the file is ~/analytics/codes.txt.
prompt: admin@server1:~$ answer: tr -d '-' < analytics/codes.txt ||| tr -d '-' < ~/analytics/codes.txt output: AX7742 CIA441 RYE093 hint: Section 13.15. The character tool with the option -d, the dash quoted as the whole set, then < and ~/analytics/codes.txt.
Three clean codes. The dash is quoted so the shell does not read it as the start of an option.
tr -d is also the usual way to remove the carriage returns a file written on Windows carries, with the set written as the escape \r.
Input from one file, output to another, in one command: < on one side and > on the other, exactly as in Section 11.5. Read ~/analytics/codes.txt and write ~/analytics/report/codes-clean.txt.
prompt: admin@server1:~$ answer: tr -d '-' < analytics/codes.txt > analytics/report/codes-clean.txt ||| tr -d '-' < ~/analytics/codes.txt > ~/analytics/report/codes-clean.txt output: hint: Section 13.15. Exercise 13-48, then > and the path ~/analytics/report/codes-clean.txt.
Silent, because both streams of the command are attached to files, and the file now holds the three codes with no dashes.
Never name the same file on both sides: the shell empties the output file before tr starts reading, and the input would be gone.
paste reads two or more files at once and writes line one of each, then line two of each, and so on, joining them with a TAB.
It is the tool for two files whose lines correspond by position: a list of times and a list of events, or the output of two commands written to two files. The two files here are ~/analytics/times.txt and ~/analytics/counts.txt. Read the times first.
prompt: admin@server1:~$ answer: cat analytics/times.txt ||| cat ~/analytics/times.txt output: 0412 0433 0515 0544 0612 hint: Section 5.1. The printing tool and the event times ~/analytics/times.txt.
Five times, one per line, and nothing else.
The second file, ~/analytics/counts.txt, holds the events themselves, in the same order as the times.
prompt: admin@server1:~$ answer: cat analytics/counts.txt ||| cat ~/analytics/counts.txt output: crate signed in oven below temperature short delivery refrigeration fault oven recovered hint: Section 5.1. The printing tool and the events ~/analytics/counts.txt.
Five events. Line one of this file belongs with line one of the times file, and so on down both. Nothing in either file says so: the position IS the relationship, which is what paste assumes.
paste takes the file names in the order the columns are to appear. Set ~/analytics/times.txt beside ~/analytics/counts.txt, times first.
prompt: admin@server1:~$ answer: paste analytics/times.txt analytics/counts.txt ||| paste ~/analytics/times.txt ~/analytics/counts.txt output: 0412 crate signed in 0433 oven below temperature 0515 short delivery 0544 refrigeration fault 0612 oven recovered hint: Section 13.16. The side-by-side tool and the two paths ~/analytics/times.txt and ~/analytics/counts.txt, times first.
Five lines, each holding a time, a TAB and an event. The order of the file names decides the order of the columns. If one file is longer than the other, paste keeps going and leaves the missing column empty.
The option -d names the character between the columns instead of a tab. A comma turns two lists into a CSV file of the kind Section 13.10 took apart. File the result in ~/analytics/report/timeline.csv.
prompt: admin@server1:~$ answer: paste -d, analytics/times.txt analytics/counts.txt > analytics/report/timeline.csv ||| paste -d, ~/analytics/times.txt ~/analytics/counts.txt > ~/analytics/report/timeline.csv output: hint: Section 13.16. Exercise 13-52 with the option that names the separator, -d,, then > and the path ~/analytics/report/timeline.csv.
Silent. Note that -d means the same thing here as it does for cut: the character between fields. Most of these tools use the same letters for the same ideas.
The result went into ~/analytics/report/timeline.csv, so read that file to see what the comma did.
prompt: admin@server1:~$ answer: cat analytics/report/timeline.csv ||| cat ~/analytics/report/timeline.csv output: 0412,crate signed in 0433,oven below temperature 0515,short delivery 0544,refrigeration fault 0612,oven recovered hint: Section 5.1. The printing tool and the file you just wrote, ~/analytics/report/timeline.csv.
A CSV file built out of two plain lists. Section 13.21 prints this same file as an aligned table.
join reads two files that share a field and prints one line for every value they have in com
Practice Chapter 13, Part 2. Text Processing: uniq, tr, paste, join and comm in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.