Learn › Linux Foundations › Arguments
Final test: Chapter 14, Arguments. Twenty-five objectives on a real Debian 12 server: xargs turning a list into arguments, batch sizes with -n, the placeholder -I {} in a sentence and inside a path, a filing sweep with cp, quoting a file name that contains a space, the null separated pipeline find -print0 into xargs -0, parallel runs with -P, and the receipt that records what actually ran. Fifty minutes, graded from the commands you run and from the machine.
This chapter covers the following topics:
What you already know: Section 11.10 joined two commands with a pipe, so the output of the left command becomes the input of the right one. Chapter 13 used that pipe with grep, sort and the other text tools. This chapter covers a limit of piping a list into a command: some programs, including cp, take file names as arguments and do not read them from standard input.
Objectives
After this chapter you can:
Use Start machine to open the real practice server. The recorded output beside each command is an example from the prepared fixture; run the command in Terminal and inspect what your machine actually reports. Commands can change its files, so check the result before continuing. A reconnect may start a new shell. Run pwd to check where you are and cd ~ to return home. Prompt colors and symbols can differ from the examples. Paths beginning ~/batch/ work from any directory; shorter batch/ alternatives assume you are in /home/admin. The chapter assessment opens on a separate fresh machine and checks the commands and results requested there.
Five questions before the material. A wrong answer costs nothing: the explanation states the fact and names the section that teaches it.
>>> A command line is the command name followed by words. Every word after the name is an argument, and cp is one of the many commands that take their work only from there. Section 14.2 covers it.
>>> A pipe delivers bytes to standard input. cp does not read file names there, so it ignores the five names and reports that its required source and destination operands are missing. Section 14.3 covers the difference.
>>> xargs reads words from its standard input and puts them on the command line of the command you name, then runs that command. It is the bridge between a list and a command that only reads arguments. Section 14.4 covers it.
>>> The shell splits a command line into words at spaces before the command ever runs. An unquoted space inside a name is therefore a word boundary, and cat is handed two names that do not exist. Section 14.8 covers it.
>>> A Linux file name may contain any character except the forward slash and the null byte. A separator that cannot appear inside a name can never be mistaken for part of one. Section 14.9 covers it.
This chapter does one job to many files at once. The server carries the material for that job in the directory ~/batch, and the file ~/batch/README.txt states what is in it. Check pwd; run cd ~ if needed. Then read it.
prompt: admin@server1:~$ answer: cat ~/batch/README.txt ||| cat batch/README.txt ||| cat /home/admin/batch/README.txt output: server1: the batch directory ============================ This directory holds a job that is done to many files at once.
file-list.txt five lines, one report name per line. It is a list of names, not the files themselves. reports/ the five reports the list names, and report-99.txt, which is a draft. The draft is in this directory and it is not on the list. filed/ empty. Copies of the listed reports belong here. labels/ three files whose names contain a space. archive/ empty. Copies of the label files belong here. notes.txt one line of text that contains an apostrophe. record.sh a program that prints how many arguments it was given and shows each one between square brackets. It also appends that same line to receipt.txt. receipt.txt one line for every run of record.sh, oldest first.
record.sh exists so that you can see what a command received. Run it and read receipt.txt to find out how many times it ran and with what. hint: Section 14.1. The reading tool from Chapter 5, aimed at ~/batch/README.txt.
Three things matter for the rest of the chapter. file-list.txt is a list of names and not the files themselves. record.sh reports what it was given, so you can see what a command actually received. receipt.txt keeps those reports, so you can count how many times a command ran.
The directory ~/batch/reports holds the files the job works on. List it.
prompt: admin@server1:~$ answer: ls ~/batch/reports ||| ls batch/reports ||| ls /home/admin/batch/reports output: report-01.txt report-03.txt report-05.txt report-02.txt report-04.txt report-99.txt hint: Section 14.1. The listing tool from Chapter 2, aimed at ~/batch/reports.
Six files. Five are weekly reports and report-99.txt is a draft. A wildcard such as reports/* would take all six, including the draft. The job in this chapter must take five, and the list is what decides which five.
A list file is an ordinary text file holding one name per line. Nothing about it is special to the shell: it is text, and it becomes useful only when something reads it. The list of this chapter is ~/batch/file-list.txt.
prompt: admin@server1:~$ answer: cat ~/batch/file-list.txt ||| cat batch/file-list.txt ||| cat /home/admin/batch/file-list.txt output: report-01.txt report-02.txt report-03.txt report-04.txt report-05.txt hint: Section 14.1. The reading tool from Chapter 5, aimed at ~/batch/file-list.txt.
Five names, one per line, and report-99.txt is not among them. The list names files but it does not contain them: every line is a piece of text that happens to look like a file name.
Section 5.7 taught wc -l, which counts lines. On a list file, the number of lines is the number of items. Count the lines of ~/batch/file-list.txt.
prompt: admin@server1:~$ answer: wc -l ~/batch/file-list.txt ||| wc -l batch/file-list.txt ||| wc -l /home/admin/batch/file-list.txt output: 5 /home/admin/batch/file-list.txt hint: Section 14.1. The counting tool from Chapter 5 with its line option -l, aimed at ~/batch/file-list.txt.
Five items. The commands in this chapter run either once with five arguments or five times with one argument. An option on the command line decides which.
The words you type after a command name are its arguments. The shell parses and expands what you typed. Unquoted whitespace separates words; quotes keep a name with spaces as one word. The shell then starts the program with the resulting argument list. The program decides what to do with them.
~/batch/record.sh is a small program that does nothing but report its own argument list: how many arguments it was given, and each one between square brackets. It also files that same line in ~/batch/receipt.txt. Give it the three words alpha, bravo and charlie.
prompt: admin@server1:~$ answer: ~/batch/record.sh alpha bravo charlie ||| batch/record.sh alpha bravo charlie ||| /home/admin/batch/record.sh alpha bravo charlie output: record: got 3 argument(s): [alpha] [bravo] [charlie] hint: Section 14.2. The path ~/batch/record.sh, then the three words alpha bravo charlie after it, separated by spaces.
Three words on the line, three arguments in the report. The square brackets mark where each argument begins and ends. A name that contains a space is one argument, and the brackets are what make that visible.
Section 11.10 taught the pipe: the output of the command on the left becomes the standard input of the command on the right. Standard input is a stream of bytes a program may choose to read. It is not the command line.
Some commands read standard input, such as wc, sort and grep. Commands such as cp, rm, mkdir, touch and echo do not turn piped words into their command-line arguments. A program may read standard input for another purpose, such as an interactive confirmation, but that still does not supply file operands. Pipe ~/batch/file-list.txt into ~/batch/record.sh and watch it receive nothing.
prompt: admin@server1:~$ answer: cat ~/batch/file-list.txt | ~/batch/record.sh output: record: got 0 argument(s): hint: Section 14.3. The reading tool and ~/batch/file-list.txt on the left of the pipe, ~/batch/record.sh alone on the right.
Zero arguments. The five names arrived on standard input and the program ran, and its command line was empty because a pipe does not write one. The next section puts those names on a command line.
xargs reads words from its standard input, builds a command line out of them, and runs the command you name with those words as its arguments. It is the bridge between a list and a command that only reads arguments.
The simplest form is xargs followed by the command to run. With no command named at all, xargs runs echo. Sending the list into xargs echo shows what the built command line looks like, because echo prints its arguments.
A command line has a maximum length, set by the system. xargs fills one command line up to that limit, runs the command, then starts another with the words that are left, so a long list produces several runs rather than one. The five-line list here fits in one. Send ~/batch/file-list.txt into the argument builder with echo.
prompt: admin@server1:~$ answer: cat ~/batch/file-list.txt | xargs echo output: report-01.txt report-02.txt report-03.txt report-04.txt report-05.txt hint: Section 14.4. ~/batch/file-list.txt on the left of the pipe, then the argument builder and the printing command echo it should run.
Five lines went in and one line came out. xargs put all five names on a single echo command line, so echo printed them separated by single spaces. The line breaks in the file are gone: xargs treats a newline as nothing more than a separator between words.
History. xargs first appeared in PWB/UNIX in 1977 and is part of the POSIX standard. The null separated pair taught in Section 14.9, find -print0 and xargs -0, was added later by the GNU project and is not in POSIX.
echo prints its arguments but it does not say how many it received. record.sh does, so putting it in place of echo turns the same pipeline into a measurement. The list is still ~/batch/file-list.txt.
prompt: admin@server1:~$ answer: cat ~/batch/file-list.txt | xargs ~/batch/record.sh output: record: got 5 argument(s): [report-01.txt] [report-02.txt] [report-03.txt] [report-04.txt] [report-05.txt] hint: Section 14.4. The same pipeline as Exercise 14-7 with ~/batch/record.sh in place of echo.
One line, so the program ran once, with five arguments. Compare it with Exercise 14-6, where the same five names produced zero arguments. The only difference between the two lines is the word xargs.
A list can turn out to hold nothing. Section 13.3 taught grep, which prints the lines of a file that match a pattern; when no line matches, it prints nothing and the pipeline that follows it reads an empty stream.
An empty stream does not stop xargs. With no words to read, xargs runs the command once anyway, with no arguments at all. Search ~/batch/file-list.txt for report-99, a name that is not on it, and send the nothing it finds to ~/batch/record.sh.
prompt: admin@server1:~$ answer: grep report-99 ~/batch/file-list.txt | xargs ~/batch/record.sh output: record: got 0 argument(s): hint: Section 14.4. The searching tool from Chapter 13 looking for report-99 in ~/batch/file-list.txt, then the pipe, the argument builder and ~/batch/record.sh.
The search printed nothing, and the recording program still ran, with zero arguments. On a command that only reports, that is harmless.
It is not harmless on a command that carries arguments of its own. A line such as ... | xargs rm -rf ~/batch/filed still runs rm -rf ~/batch/filed when the input is empty, because the arguments you typed are on the command line whether or not xargs read anything. That is what the next option is for.
The -r option tells xargs to run nothing when its input holds no words. The long form is --no-run-if-empty. It is a GNU option, and it is the safe habit on any pipeline whose left side may find nothing. Run the same search again with it.
prompt: admin@server1:~$ answer: grep report-99 ~/batch/file-list.txt | xargs -r ~/batch/record.sh output: hint: Section 14.4. The same line as Exercise 14-9 with the option -r, which tells the argument builder not to run on nothing.
Nothing at all this time. The command was never started, so there is no line to read and no line in the receipt. Compare the two exercises: the same empty input, and one option decides whether the command runs once or not at all.
By default xargs puts as many words as it can on one command line. The -n option sets the maximum number of arguments per run, and xargs then runs the command as many times as it needs to. Run ~/batch/record.sh once per name of ~/batch/file-list.txt.
prompt: admin@server1:~$ answer: cat ~/batch/file-list.txt | xargs -n 1 ~/batch/record.sh output: record: got 1 argument(s): [report-01.txt] record: got 1 argument(s): [report-02.txt] record: got 1 argument(s): [report-03.txt] record: got 1 argument(s): [report-04.txt] record: got 1 argument(s): [report-05.txt] hint: Section 14.5. The batch size option -n set to 1, running ~/batch/record.sh over ~/batch/file-list.txt.
Five lines, so the program ran five times, each time with one argument. The number of lines is the number of runs. Anything that must happen once per file, such as a copy or a rename, needs a batch size of one.
-n is a maximum, not a fixed size. When the items run out, the last run gets whatever is left. Send the same list two names at a time.
prompt: admin@server1:~$ answer: cat ~/batch/file-list.txt | xargs -n 2 ~/batch/record.sh output: record: got 2 argument(s): [report-01.txt] [report-02.txt] record: got 2 argument(s): [report-03.txt] [report-04.txt] record: got 1 argument(s): [report-05.txt] hint: Section 14.5. The same batch size option with the number 2.
Three runs from five items: two, two, and one left over. xargs never pads a batch and never drops the remainder.
So far the names have been appended to the end of the command line. When the command needs them somewhere else, the -I option names a placeholder: a short piece of text that xargs replaces with the item it read. The usual placeholder is {}, written after -I.
-I also changes how the input is split. With it a space no longer ends an item: only a newline does, so one whole line is one item and the command runs once for each line. With nothing at all to read, -I runs the command no times. Put each name of ~/batch/file-list.txt in the middle of a sentence.
prompt: admin@server1:~$ answer: cat ~/batch/file-list.txt | xargs -I {} echo filing {} into the archive output: filing report-01.txt into the archive filing report-02.txt into the archive filing report-03.txt into the archive filing report-04.txt into the archive filing report-05.txt into the archive hint: Section 14.6. Read ~/batch/file-list.txt in first, then the option -I with the placeholder {}, and a printing command whose words surround the placeholder: filing {} into the archive.
Five runs again, and this time the name sits between the words filing and into. Without -I the five names would have been appended after archive. The placeholder is what decides the position.
The placeholder is replaced wherever it appears, including inside a longer piece of text such as a path. That turns a bare name on the list into a full path the command can use. Count the lines of every report named in ~/batch/file-list.txt, which live in ~/batch/reports.
prompt: admin@server1:~$ answer: cat ~/batch/file-list.txt | xargs -I {} wc -l ~/batch/reports/{} output: 2 /home/admin/batch/reports/report-0
Practice Chapter 14. Arguments: xargs, placeholders and null separated names in a real Linux terminal at The Linux Camp. Progress is verified automatically as you type commands on the machine.