Learn › Linux Foundations › I/O Redirection
pipeline - a hands-on Linux lab on a real virtual machine.
Chain two or more pipes into a pipeline: sort and dedupe a stream, pull the largest value with a numeric sort, tally repeats with uniq -c, and filter mid-stream before counting, then rebuild three of the chains from memory in the practice terminal.
In the last lesson you wired two commands together with a single pipe: cat fruits.txt | wc -l counted five lines with no temporary file in sight. One pipe, one transformation.
But real questions usually take more than one transformation. Here is one now. The numbers.txt file in your signal folder is a mess: the numbers are out of order, and at least one of them appears twice. You want a clean list, every number in order, each one exactly once.
Linux has a tool for each half of that job. sort puts lines in order. uniq folds duplicate lines into one. Both get their own full lessons later in the camp; today they are stage tools. The problem: neither one can do the whole job alone.
The black boxes below are a practice terminal: a safe sandbox that only checks the one command each step teaches, so you cannot break anything. You are working inside the signal folder in your home directory, the small fixture folder this module uses. On the real machine at the end of the module, the capstone mission builds this same signal folder and you run these moves for real.
First, look at the raw data with your own eyes:
prompt: student@linuxcamp:~/signal$ answer: cat numbers.txt output: 3 1 4 1 5 9 hint: Print the file to the screen: cat numbers.txt
Six lines, scrambled, and the 1 shows up twice. Cleaning this takes two different transformations on one stream of text: put it in order AND fold the duplicate. A single pipe gives you one hand-off. This lesson gives you the rest: the pipeline.
A pipeline is two or more pipes in a row: command | command | command. The text flows left to right. Each stage reads the stream on its stdin (channel 0), reshapes it, and hands the result to the next stage on its stdout (channel 1). Only the last stage prints to your screen.
Nothing new happens at each |. It is the exact same hand-off you learned in the pipe lesson, just repeated. What is new is the way of thinking: you stop hunting for one command that does everything and start snapping small commands together, each doing one small job.
That mindset has a name. In 1978 Doug McIlroy, the Bell Labs manager who pushed pipes into Unix, wrote it down as the Unix philosophy: make each program do one thing well, and expect the output of every program to become the input of another. A pipeline is that sentence turned into syntax.
Time to clean numbers.txt. The chain is cat numbers.txt | sort | uniq: cat produces the raw lines, sort puts them in order, and uniq folds repeated neighbors into one. Watch the stream travel through the stages, one hop at a time:
{ "caption": "cat numbers.txt | sort | uniq: each pipe hands the stream to the next stage, and only the last stage prints to your screen.", "nodes": [ { "id": "cat", "label": "cat numbers.txt", "kind": "command", "col": 0, "row": 0 }, { "id": "sort", "label": "sort", "kind": "command", "col": 1, "row": 0 }, { "id": "uniq", "label": "uniq", "kind": "command", "col": 2, "row": 0 }, { "id": "screen", "label": "screen", "kind": "screen", "col": 3, "row": 0 } ], "edges": [ { "from": "cat", "to": "sort", "label": "| unsorted", "kind": "stdout", "stage": 1 }, { "from": "sort", "to": "uniq", "label": "| sorted", "kind": "stdout", "stage": 2 }, { "from": "uniq", "to": "screen", "label": "unique lines", "kind": "stdout", "stage": 3 } ] }
One detail before you run it. uniq only folds duplicates that sit on neighboring lines. In the raw file the two 1s are separated, so uniq alone would miss them. sort parks them side by side first. That is why the stage order is sort | uniq and never the other way around.
Before you run this, decide: six lines go in, so how many will come out?
prompt: student@linuxcamp:~/signal$ answer: cat numbers.txt | sort | uniq output: 1 3 4 5 9 hint: Chain three stages with two pipes: cat numbers.txt | sort | uniq
Five. Count them against the six that went in: the second 1 is gone. Read the pipeline left to right like a story. cat produced the scrambled lines, sort reordered the stream so the two 1s sat side by side, and uniq folded those neighbors into one. Each | is a hand-off, each stage does one small job, and you just ran two transformations on one stream with zero temporary files.
New question, same file: what is the largest number in numbers.txt? No stage in the last chain answers that, so you swap stages. This is the real skill of pipelines: choosing stages to fit the job.
Two swaps. First, sort -n: the -n flag makes sort compare lines as numbers instead of text. On single digits plain sort happens to land in the same order, but on bigger values text order goes wrong (as text, 10 sorts before 9), so reach for -n whenever the lines are numbers. Second, tail -1: remember tail from the reading lessons, it shows the end of its input, and -1 keeps just the final line. After a numeric sort, the final line is the biggest value.
Before you run this, decide exactly which single number will print.
prompt: student@linuxcamp:~/signal$ answer: cat numbers.txt | sort -n | tail -1 output: 9 hint: Sort numerically, then keep one line from the end: cat numbers.txt | sort -n | tail -1
9, the maximum, and nothing else. The pipeline reads like a sentence: print the file, sort it as numbers, keep the last line. Notice you did not learn a max command, because there is none to learn. You composed one out of stages you already had. Swap tail -1 for head -1 and the same chain would hand you the smallest number instead.
colors.txt holds four lines: red, green, green, blue. This time you do not want the duplicates removed, you want them counted: how many times does each color appear?
One flag does it. uniq -c folds repeated neighbors exactly like plain uniq, and prefixes each surviving line with a count of how many lines it folded together. Same chain as before, one flag added.
Before you run this, decide: which color will wear the 2?
prompt: student@linuxcamp:~/signal$ answer: cat colors.txt | sort | uniq -c output: 1 blue 2 green 1 red hint: The tally chain is the dedupe chain plus one flag: cat colors.txt | sort | uniq -c
There is the tally: green wears the 2 because it appeared on two lines, and the counts landed in a new column on the left. The extra spaces in front of each count are normal, uniq -c pads the numbers so they line up in a right-aligned column. Now look at what changed from the last dedupe: one flag turned remove-the-duplicates into count-the-duplicates. sort still had to run first, because uniq -c also only counts neighboring repeats.
One more stage type completes the set: a filter. grep keeps only the lines that match a pattern and drops the rest; it gets a full lesson later in the camp. Here grep e keeps the lines that contain the letter e.
The question: how many fruit names in fruits.txt contain an e? Filter first, then count with wc -l, the line counter you already know.
Before you run this, decide on the number. The five fruits are apple, banana, cherry, date, and elderberry, and banana has no e.
prompt: student@linuxcamp:~/signal$ answer: cat fruits.txt | grep e | wc -l output: 4 hint: Keep the matching lines, then count them: cat fruits.txt | grep e | wc -l
4: apple, cherry, date, and elderberry made it through the filter, banana did not, and wc -l counted what survived. Stop and notice what just happened. You answered a question about the data without ever reading the data yourself. The pipeline read it, filtered it, and measured it for you. That is the milestone of this lesson: you are not running commands one at a time anymore, you are designing little factories. Produce, reshape, measure. From here the scaffolding comes off.
Scaffolding off. No command is shown from here on.
From numbers.txt, print every number in order with each one appearing exactly once. Three stages, two pipes.
prompt: student@linuxcamp:~/signal$ answer: cat numbers.txt | sort | uniq output: 1 3 4 5 9 hint: Print the file into the pipe, order the lines, then fold the neighboring duplicates.
Five clean lines, from memory. Ordering stage before folding stage, because the folder only sees neighbors. That chain is a daily workhorse on real systems, from log files to inventory lists.
Still no command shown. Make the terminal print only the largest number in numbers.txt. Remember which flag makes the ordering stage treat lines as numbers, and which stage keeps one line from the end.
prompt: student@linuxcamp:~/signal$ answer: cat numbers.txt | sort -n | tail -1 output: 9 hint: Order the stream numerically, then keep just the final line.
9, alone. You built a max finder out of three small tools, and you could rebuild it tomorrow without notes. That is stage thinking doing its job.
Last one, from memory. Print a single number: how many lines of fruits.txt contain the letter e. A filter stage, then a counting stage.
prompt: student@linuxcamp:~/signal$ answer: cat fruits.txt | grep e | wc -l output: 4 hint: Keep only the lines with an e in them, then count what comes through.
4, exactly as before, and this time the chain came out of your own head. Three challenges, three pipelines, zero temporary files.
You earned this cheat sheet. Every chain in it is one you built and ran in the signal folder:
The one idea under all of it: a pipeline is the single pipe, repeated. Each stage reads the stream on stdin (channel 0), does one small job, and writes the result on stdout (channel 1) for the next stage. Order your stages so each one gets the input it needs, the way sort feeds uniq its neighbors.
Build long chains one stage at a time. Run the first command alone, look at the output, then add the next | stage and run again. Every stage boundary is a place you can pause and inspect the stream, which is exactly how working engineers debug pipelines too.
You went from one pipe to full pipelines: a three-stage dedupe, a max finder, a tally, and a filter-then-count, then three of them again from memory. Chains of small tools are how a huge amount of real Linux work gets done.
Two more moves complete this module. tee saves a copy of the stream to a file without stopping the flow, perfect for snapshotting the middle of a pipeline. Combining redirects mixes the file operators with pipes in one command.
The io-redirection module ends with one real Linux machine: the redirection mission lab. There a VM boots just for you with this same signal folder, and the mission hands you objectives with no commands shown. You read the objective, recall the chain, and type it stage by stage. That recall is what makes it stick.
Finish the module's lessons, then go build some pipelines for real.
Practice Pipe Chains: Build a Pipeline in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.