xargs -P: run jobs in parallel

xargs - a hands-on Linux lab on a real virtual machine.

Run xargs jobs in parallel with -P: up to N command runs at once. See why the on-screen order varies, prove a parallel run with sort and wc -l, shape run sizes with -n2, and combine the -0 safety pair with -P4 for safe parallel work on space-named files. All in the practice terminal.

You have spent three lessons teaching xargs to work for you. It turns text on stdin into arguments, -n shapes the batches, and the -print0 with -0 pair carries space-named files safely. Through all of it, one thing never changed: every command run happened one after another, in a queue.

Today the queue breaks open, and something strange comes with it. There is an xargs option that makes the same command print its results in a DIFFERENT order from one run to the next. Same command, same file, two runs, two different screens. Nothing is broken, and both screens are correct. By the end of this lesson you will know why, and you will know the two ways to prove a run like that still did its job.

The black boxes below are a practice terminal: a safe sandbox that only checks the one command each step teaches, so you cannot break anything. Everything runs from inside ~/args-base, the module fixture: words.txt holds four words (alpha, bravo, charlie, delta, one per line) and the files/ folder holds five files. On the real machine at the end of the module, the Arguments mission builds this same folder and you run these commands for real.

First, feel the queue itself. Remember from the first xargs lesson: -n1 gives each run at most one argument, so four words mean four separate runs of echo. Before you run it, commit to an answer: in what order will the four words print, and could that order ever change between runs?

prompt: student@linuxcamp:~/args-base$ answer: cat words.txt | xargs -n1 echo output: alpha bravo charlie delta hint: One argument per run, like the first xargs lesson: cat words.txt | xargs -n1 echo

File order, top to bottom: alpha, bravo, charlie, delta. And no, it cannot change. xargs starts run one, waits for it to finish, then starts run two. Each run ends before the next begins, so the output order always matches the input order. That guarantee is so quiet you never noticed it. It is also a speed limit: while one run works, the other three wait in line.

The -P option (a capital P) lifts that speed limit. -P4 tells xargs it may keep up to FOUR command runs going at the same time. When the jobs are independent of each other, compressing files, converting images, checking a list of addresses, that can cut the total time dramatically, because a modern machine really can do several things at once.

Here is the price. Parallel runs finish whenever they finish, and whichever finishes first prints first. So the ORDER of the output lines is no longer promised: run the same -P command twice and the lines can land in a different sequence each time. That is the mystery from the top of the lesson. It is not a bug. Unpredictable order is simply what "at the same time" means.

xargs has been building argument lists since the 1970s, when it shipped with the Unix Programmer's Workbench. The parallel -P switch is a later GNU findutils addition, and the idea proved so useful it grew into an entire dedicated tool, GNU parallel.

One thing must be clear before you go parallel: -P does not run your WORDS at the same time, it runs COMMAND RUNS at the same time. The run is the unit of parallel work. And you already own the dial that decides how many runs there are: -n, the batch size.

-n1 made four words into four runs. -n2 packs TWO arguments into each run instead. Before you run it, decide: how many times will echo run on the four words, and which two words will share the first line?

prompt: student@linuxcamp:~/args-base$ answer: cat words.txt | xargs -n2 echo output: alpha bravo charlie delta hint: Same shape as -n1, but with a batch size of two: cat words.txt | xargs -n2 echo

Two lines: echo ran twice, two words each time, still in file order because nothing is parallel yet. Now connect it to -P. With -n2 there are two runs, so -P2 could put BOTH in flight at once. -n decides how the work splits into runs; -P decides how many of those runs may go at the same time. The two options are teammates, and you will almost always see them side by side.

Time to break the queue. -P4 -n1 means four runs, all allowed to go at once. But you just learned the raw output order of a parallel run cannot be trusted, and a result you cannot check is a result you cannot use. The escape is one more pipe: sort reads all the lines and prints them in a fixed order, no matter which run finished first.

Before you run it, decide: after the sort, will you see the same four words as your sequential baseline, or something different?

prompt: student@linuxcamp:~/args-base$ answer: cat words.txt | xargs -P4 -n1 echo | sort output: alpha bravo charlie delta hint: The -n1 pipeline with -P4 added, then a sort on the end: cat words.txt | xargs -P4 -n1 echo | sort

Only that final | sort makes this output the same every time. Without it, the four words would still all appear, but in whatever order the four parallel runs happened to finish, which varies on every run and on every machine. Never judge a -P command by its raw on-screen order.

Hold this next to the baseline from the first step: alpha, bravo, charlie, delta. Identical, line for line. That sameness IS the finding. Parallelism changed WHEN each line printed, never WHAT was printed: the SET of results survived untouched. Pause on that for a second. Three lessons ago a bare pipe could not hand echo a single word. You just launched four jobs at once and proved the result, and that habit is exactly what keeps parallel pipelines honest in real work.

Scaffolding off. No command is shown from here on.

The set, proved with sort, is one stable fact of a parallel run. The other is the COUNT: four inputs produce four output lines no matter how the runs interleave. Order moves, the line count never does. Prove it. Run the four-wide parallel echo on words.txt again, but this time count its output lines instead of sorting them. You already own the counting tool: it is the one that exposed the broken token count of 7 two lessons ago.

prompt: student@linuxcamp:~/args-base$ answer: cat words.txt | xargs -P4 -n1 echo | wc -l output: 4 hint: The same -P4 -n1 pipeline as the sorted drill, but swap the sort for the line counter

Four, every single time, however the runs shuffle. Interleaving changes the order lines appear in, never how many there are. You now hold both proofs: sort shows the set is right, wc -l shows the size is right. Those are the only two things worth checking about parallel output, because they are the only two things it promises.

One more, from memory, and it chains the entire module. Print every file in the files/ folder, one per run, with up to four runs at once, and force the output into a stable sorted order. Remember the trap: two of those names carry a space, and remember from the -0 lesson which pair of options carries space-names through a pipe safely.

prompt: student@linuxcamp:~/args-base$ answer: find files -type f -print0 | xargs -0 -P4 -n1 echo | sort output: files/data.csv files/final draft.txt files/notes.txt files/q1 report.txt files/report.txt hint: Start from the NUL-safe find pipeline, add the parallel flag next to -n1 echo, and finish with a sort

Five lines for five files, both space-names whole, order fixed. Look at what each piece bought you: -print0 with -0 kept files/final draft.txt and files/q1 report.txt in one piece, -P4 ran the jobs four wide, and sort made the result checkable. Safety, speed, and proof in a single pipeline. That is every lesson of this module in one line, typed from memory.

You earned this cheat sheet. Every row is a command you ran on words.txt or the files/ folder in this lesson:

The one idea that ties it together: -P N lets xargs keep up to N command runs going at the same time. The speed costs you the on-screen order, so never judge a parallel run by how it scrolls. Prove the set with sort, prove the size with wc -l, and shape the runs themselves with -n.

The default is -P1, which is exactly the one-after-another behavior you have always had, and -P0 tells xargs to run as many at once as it can manage. A sensible starting value for real work is the number of processor cores your machine has: more runs than cores just means the runs take turns anyway.

This lesson gave xargs its last gear. The queue is the default and its order is guaranteed. -n shapes the input into runs, -P lets several runs fly at once, and a parallel run is judged by its set (sort) and its count (wc -l), never by its raw order. Every one of those you typed yourself and saw the real result.

The Arguments module ends with one real Linux machine: the Arguments mission lab. There a VM boots just for you on this same words.txt and files/ fixture, with xargs and find ready to go. The mission hands you objectives that use exactly what you just ran, it shows no commands, and its final objective is the very proof you built here: confirm the count holds under -P. You read the objective, recall the option, and type it. That recall is what makes it stick.

Go run the assembly line four wide.

Practice xargs -P: run jobs in parallel in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.