xargs

xargs - a hands-on Linux lab on a real virtual machine.

The first xargs lesson: the bridge from stdin to arguments. Feed xargs words from echo and cat, watch the default join everything into one run, hand items over one at a time with -n1, and drop each item into place with -I{}. The whitespace trap, the NUL fix, and parallel -P each get their own lesson next. All in the practice terminal.

Here is a small puzzle that trips up almost everyone new to Linux. You have a list of things sitting in a file, or streaming out of another command, and you want to run a command on that list. So you try the obvious thing and pipe the list in. Nothing happens, or the wrong thing happens. The list went in, but the command ignored it.

The reason is that many commands do not read their work from a pipe at all. They read it from their arguments: the words you type after the command name, like the hero.jpg in rm hero.jpg. A pipe feeds a command's standard input (its stdin), a separate channel. Give rm a filename on stdin and it just waits, because rm never looks there. It only looks at its arguments.

xargs is the bridge between those two worlds. It reads a list from stdin and turns that list into arguments for a command it runs for you. In this lesson you build that bridge and steer it three ways: everything in one run, one item per run, and each item dropped into the exact spot you choose.

The black boxes below are a practice terminal: a safe sandbox that only checks the one command each step teaches, so you cannot break anything. On the real machine at the end of the module, the mission builds this same args-base folder and you run these commands for real. Everything here runs from inside ~/args-base, which holds a file words.txt and a folder files.

Start with the simplest possible case. When you run xargs with no command after it, it quietly uses echo as the default, and echo does nothing but print the arguments it is given. That makes bare xargs a little x-ray machine: whatever it prints is exactly the argument list it built from your stdin.

Hand it three words down a pipe. Before you run it, decide: will the three words come back on one line, or on three separate lines?

echo "alpha bravo charlie" | xargs

prompt: student@linuxcamp:~/args-base$ answer: echo "alpha bravo charlie" | xargs output: alpha bravo charlie hint: Type the echo, then a padded pipe, then xargs. Like this: echo "alpha bravo charlie" | xargs

One line. The three words that arrived on stdin became three arguments to a single run of echo, which printed them side by side. To see why that is remarkable, picture the same pipe without the bridge. echo "alpha bravo charlie" | echo prints an empty line, because the right-hand echo ignores stdin and had no arguments to print. xargs is exactly the piece that carried the words across from stdin to the argument list. That is its whole job.

xargs has been doing that job since the 1970s. It was born because the kernel limits how much text fits on one command line: hand the shell tens of thousands of filenames at once and it gives up with Argument list too long. xargs was written to chop a giant list into batches that fit, and it has been the standard bridge from stdin to arguments ever since.

The list rarely comes from a literal string you typed. Much more often it streams out of another command. The fixture file words.txt holds four words, one per line: alpha, bravo, charlie, delta. Print it onto stdout with cat and pipe that into the bridge.

Before you run it, decide: the four words sit on four separate lines. How many lines will come back?

cat words.txt | xargs

prompt: student@linuxcamp:~/args-base$ answer: cat words.txt | xargs output: alpha bravo charlie delta hint: Print the file, then a padded pipe, then xargs. Like this: cat words.txt | xargs

One line, not four. xargs collected all four tokens and ran echo once, joining them onto a single line. That is the default behavior: gather every token from stdin, hand them all to one run of the command. Notice it treated the newlines as gaps between tokens, exactly like spaces. And the echo is implied: cat words.txt | xargs echo prints the exact same line. Next you make xargs stop cramming everything into one run.

By default xargs piles every token onto one command line. The -n option changes that: -n1 means "pass at most one argument to each run," so the command runs once per token instead of once total.

Point it at the same four words. Before you run it, decide: how many times will echo run, and how many lines will you see?

cat words.txt | xargs -n1 echo

prompt: student@linuxcamp:~/args-base$ answer: cat words.txt | xargs -n1 echo output: alpha bravo charlie delta hint: Add -n1 before echo so each word is its own run: cat words.txt | xargs -n1 echo

Four runs, four lines, one word each. Hold this next to the previous drill: same four words, same pipe, but -n1 turned one long line into four short ones, because echo now fired once per token. The number is yours to pick. -n2 would pass two arguments per run, printing alpha bravo on one line and charlie delta on the next: two runs of two. You will put that batch dial to real work in the parallel lesson at the end of this module.

So far the arguments always landed at the end of the command. Sometimes you need each item dropped into the middle of a command instead. The -I option sets up a placeholder: you pick a marker, and xargs runs the command once per input line, substituting that line wherever your marker appears. The usual marker is {}.

Try it as a labeler: -I{} with the command echo item: {}. Before you run it, decide: four input lines again, so four lines out, but where in each line will the word land?

cat words.txt | xargs -I{} echo item: {}

prompt: student@linuxcamp:~/args-base$ answer: cat words.txt | xargs -I{} echo item: {} output: item: alpha item: bravo item: charlie item: delta hint: Set the placeholder with -I{} and reuse {} in the command: cat words.txt | xargs -I{} echo item: {}

Each word slotted into the middle of the command, right where {} sat. The command ran once per input line, because -I implies one line at a time, so you did not need -n1 here. This is the form you reach for when the item is not the last word of the command, for example xargs -I{} cp {} backup/ to copy each listed file into a backup folder. Placeholder in, command runs, item lands exactly where you put the marker.

Stop and look at what you can already do. At the start of this lesson, a pipe full of words was useless to any command that reads only arguments. Now you can take a list on stdin and aim it at a command three different ways: all at once, one per run with -n1, or dropped into a template with -I{}. That is the core of xargs. Everything else in this module is refinement.

Three refinements are coming, and each gets its own lesson. A warning first: every token you fed the bridge today was a clean single word. Inside the files folder sit two filenames with a space in the middle, and one space inside a name is enough to tear a naive xargs pipeline apart. That trap gets its own lesson next. The fix, a separator no filename can ever contain, gets the lesson after that. And -P, which lets xargs run several commands at the same time, closes out the module.

You have every form you need. These three challenges show you only the goal. Recall the command yourself; that recall is what makes it stick.

Challenge 1. Join the four words of words.txt onto a single line, letting the bridge use its default command.

prompt: student@linuxcamp:~/args-base$ answer: cat words.txt | xargs output: alpha bravo charlie delta hint: Read the file out with cat, then send it across the bridge with no options. The default gathers everything into one run.

Challenge 2. Without touching words.txt, send the three words alpha bravo charlie down the pipe as one double-quoted string, and print each word on its own line.

prompt: student@linuxcamp:~/args-base$ answer: echo "alpha bravo charlie" | xargs -n1 echo output: alpha bravo charlie hint: Print the three words inside double quotes first, then cap the bridge at one argument per run so each word gets its own line.

Challenge 3. Turn words.txt into a labeled list: four lines reading item: alpha down to item: delta, with each word dropped into place after the label.

prompt: student@linuxcamp:~/args-base$ answer: cat words.txt | xargs -I{} echo item: {} output: item: alpha item: bravo item: charlie item: delta hint: This is placeholder territory. Pick {} as your marker and put the marker exactly where each word should land.

You earned this cheat sheet. Every row is a command you just ran on the fixture:

The one idea to carry out of this lesson: xargs builds an argument list from stdin. The default joins everything into one run, -n sets how many items each run gets, and -I{} places each item exactly where you want it inside the command.

Before you ever point an xargs pipeline at a destructive command like rm, run the pipeline with no command at all first. The default echo prints exactly the argument list the real command would receive, which makes bare xargs a free dry run.

You built the stdin-to-arguments bridge and steered it three ways: the default join, one item per run with -n1, and template placement with -I{}. Every one of those you typed yourself and saw the real result.

The Arguments module ends with one real Linux machine: the arguments mission lab. There a VM boots just for you and builds this same args-base fixture, words.txt and the files folder included. The mission hands you objectives drawn from all four xargs lessons, and it shows no commands. You read the objective, recall the form, and type it. That recall is what makes it stick.

Finish the other Arguments lessons, then go run it for real.

Practice xargs in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.