Learn › Linux Foundations › Text Processing

uniq

uniq - a hands-on Linux lab on a real virtual machine.

Collapse repeated lines with uniq. The one rule that trips everyone: uniq only compares adjacent lines, so you sort first. Count with -c, split repeats from loners with -d and -u, ignore case with -i, all in the practice terminal on two tiny fixtures.

You have a file full of names, and some of them repeat. You do not want to read every line; you want the DISTINCT ones, each name once, however many times it showed up. There is a command built for exactly that, and it is called uniq.

But uniq has one habit that surprises almost everyone the first day. It does not find every duplicate in a file. It only collapses duplicates that sit RIGHT NEXT TO each other. Get that one rule and uniq is simple. Miss it and you will swear the command is broken. This lesson is about that rule and the handful of flags that ride on top of it.

The black boxes in this lesson are a practice terminal: a safe sandbox that only checks the one command each step teaches, so you cannot break anything. At the end of the module one real Linux machine boots just for you and you run the full mission there.

You have a small file called sorted.txt. It holds six lines, already in order:

apple
apple
banana
cherry
cherry
cherry

Two apples, one banana, three cherries. Point plain uniq at it and press Enter:

uniq sorted.txt

prompt: student@linuxcamp:~$ answer: uniq sorted.txt output: apple banana cherry hint: Type uniq, a space, then the filename: uniq sorted.txt

Six lines went in, three came out. uniq walked the file top to bottom, and every time a line matched the one directly before it, it threw the copy away. The two apples became one apple, the three cherries became one cherry. What you are left with is each name once, in the order it first appeared. That is uniq in its plainest form.

uniq is short for unique. It comes from the early Unix toolkit, where the design idea was many small commands that each do one job and hand their output to the next. uniq's one job is de-duplication: take a stream of lines and squeeze out the repeats.

The reason it only looks at neighbouring lines is that same design. uniq reads the file once, straight through, holding only the current line and the one before it in mind. It never scans the whole file to hunt for a match somewhere far away. That makes it fast and tiny, and it is why the input almost always needs to be sorted first, which you will see in a moment.

Collapsing repeats is useful, but often you want to know HOW MANY there were. The -c flag, short for count, prints the number of times each line appeared, right in front of the line.

Run it on the same sorted.txt:

uniq -c sorted.txt

prompt: student@linuxcamp:~$ answer: uniq -c sorted.txt output: 2 apple 1 banana 3 cherry hint: Add -c between uniq and the filename: uniq -c sorted.txt

Now each line carries its tally. 2 apple because apple appeared twice, 1 banana because banana appeared once, 3 cherry because cherry appeared three times. The number is padded with spaces so the counts line up in a neat column, which is why you see the leading spaces before each digit. uniq -c is the everyday way to answer "how many of each?" once a list is sorted.

Two more flags split the list the opposite way from each other. -d, short for duplicated, prints ONLY the lines that repeat, one copy each, and hides everything that appeared just once. Its mirror, -u, short for unique in the strict sense, prints ONLY the lines that appeared exactly once and hides every repeat.

Ask sorted.txt for just the lines that repeat:

uniq -d sorted.txt

prompt: student@linuxcamp:~$ answer: uniq -d sorted.txt output: apple cherry hint: Use -d for duplicated: uniq -d sorted.txt

Apple and cherry, because those are the two names that showed up more than once. Banana is gone: it appeared exactly once, so -d drops it. Now flip it. Run uniq -u sorted.txt and you get the exact opposite, just banana, because it is the only line that never repeated. -d keeps the repeats, -u keeps the loners. Between them they cut the same list two different ways.

This is the mistake every uniq user makes once, and it is far better to meet it here in the sandbox. Everything so far worked because sorted.txt was already sorted, so every duplicate sat next to its twin. Watch what happens when the duplicates are scattered.

There is a second file, raw.txt, with the same kind of names but NOT sorted. It holds four lines:

apple
banana
apple
cherry

Apple appears twice, but there is a banana sitting between the two apples. Run plain uniq on it and watch:

uniq raw.txt

prompt: student@linuxcamp:~$ answer: uniq raw.txt output: apple banana apple cherry hint: Type uniq and the unsorted filename: uniq raw.txt

Nothing collapsed. All four lines came back, apple twice. uniq compared each line only to the one directly before it, and the two apples were never neighbours (a banana sat between them), so it never saw them as a pair. This is not a bug. It is the whole rule: uniq de-duplicates ADJACENT lines and nothing else. When uniq seems to miss obvious duplicates, the input was not sorted.

The cure is to make the duplicates adjacent before uniq ever sees them, and there is a command for that: sort, which rearranges a file's lines into order. sort has its own lesson; here you only need that it pulls matching lines together so they sit side by side.

You join the two commands with a |, called a pipe, which hands the output of the command on its left straight into the command on its right. So sort raw.txt | uniq means: sort the lines first, then feed that sorted stream into uniq. Run it:

sort raw.txt | uniq

prompt: student@linuxcamp:~$ answer: sort raw.txt | uniq output: apple banana cherry hint: sort the file, a pipe, then uniq: sort raw.txt | uniq

Three clean lines. sort pulled the two apples next to each other, and only then could uniq see them as a pair and collapse them. This pairing, sort file | uniq, is so common it is almost a single reflex. Whenever you want the distinct values out of any list, you sort first so the duplicates line up, then let uniq squeeze them. Remember the order: sort, then uniq.

By default uniq is picky about capitals. To it, Apple and apple are two different lines, because one byte differs. The -i flag, short for ignore case, tells uniq to treat upper and lower case as the same, so Apple, APPLE, and apple all count as one line.

You will reach for -i any time the capitalisation in a list is inconsistent but the words are really the same. It sits alongside the others: uniq -ci counts while ignoring case, uniq -di shows repeats while ignoring case. The rule underneath is unchanged: the lines still have to be adjacent, so you still sort first.

Scaffolding off. No command is shown this time. You have every piece you need.

You are handed raw.txt again, the unsorted one with apple, banana, apple, cherry. You want a tally of how many times each name appears, one count per name. Remember what bit you earlier: uniq only sees neighbours, so you must line the duplicates up before you count them. Then ask uniq for the count.

prompt: student@linuxcamp:~$ answer: sort raw.txt | uniq -c output: 2 apple 1 banana 1 cherry hint: First sort so duplicates are adjacent, then count with -c: sort raw.txt | uniq -c

Two apple, one banana, one cherry, from memory. You had to remember the trap and beat it: sort first so the two apples became neighbours, then uniq -c to tally them. Skip the sort and the count would have been wrong, splitting apple into two separate ones. That is the whole skill of uniq: get the lines adjacent, then collapse or count. It is exactly the move the real machine will ask of you.

You earned this cheat sheet. Every row is a form of uniq you have already run:

The one rule to carry away: uniq only compares each line to the one directly before it. Duplicates that are not neighbours are invisible to it, which is why you almost always sort the file first. Sort to line them up, then uniq to squeeze or count them.

The sort file | uniq pattern is so standard that sort even has its own -u flag that does both in one step. You do not need it yet; knowing WHY the pair exists (adjacency) matters more than the shortcut, and the shortcut waits in the sort lesson.

The practice terminal has shown you every form of uniq: the plain collapse, -c to count, -d and -u to split repeats from loners, -i to ignore case, and the adjacency trap that the sort file | uniq pairing exists to solve. Every one of those you typed yourself.

The Text Processing module ends with one real Linux machine, the text-processing capstone mission, and that is where you de-duplicate for real. A VM boots just for you, with its own files to sift. Its objectives use exactly what you practiced across this module, including uniq alongside its neighbours like sort, cut, and grep, each of which has its own lesson. One difference from here: the mission shows no commands. You read the objective, recall the command, and type it. That recall is what makes it stick.

Finish the other Text Processing lessons, then go collapse a real list for yourself.

Practice uniq in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.