When is xargs needed?

Question

The xargs command always confuses me. Is there a general rule for it?

Consider the two examples below:

$ \ls | grep Cases | less

prints the files that match 'Cases', but changing the command to touch will require xargs:

$ \ls | grep Cases | touch
touch: missing file operand
Try `touch --help' for more information.

$ \ls | grep Cases | xargs touch

Answer 1

The difference is in what data the target program is accepting.

If you just use a pipe, it receives data on STDIN (the standard input stream) as a raw pile of data that it can sort through one line at a time. However some programs don't accept their commands on standard in, they expect it to be spelled out in the arguments to the command. For example touch takes a file name as a parameter on the command line like so: touch file1.txt.

If you have a program that outputs filenames on standard out and want to use them as arguments to touch, you have to use xargs which reads the STDIN stream data and converts each line into space separated arguments to the command.

These two things are equivalent:

# touch file1.txt
# echo file1.txt | xargs touch

Don't use xargs unless you know exactly what it's doing and why it's needed. It's quite often the case that there is a better way to do the job than using xargs to force the conversion. The conversion process is also fraught with potential pitfalls like escaping and word expansion etc.

Answer 2

To expand on the answers already provided, xargs can do one cool thing that is becoming increasingly important in today's multicore and distributed computing landscape: it can parallel process jobs.

For example:

$ find . -type f -name '*.wav' -print0 |xargs -0 -P 3 -n 1 flac -V8

will encode *.wav => *.flac, using three processes at once (-P 3).

Answer 3

xargs is particularly useful when you have a list of filepaths on stdin and want to do something with them. For example:

$ git ls-files "*.tex" | xargs -n 1 sed -i "s/color/colour/g"

Let's examine this step by step:

$ git ls-files "*.tex"
tex/ch1/intro.tex
tex/ch1/motivation.tex
....

In other words, our input is a list of paths that we want to do something to.

To find out what xargs does with these paths, a nice trick is to add echo before your command, like so:

$ git ls-files "*.tex" | xargs -n 1 echo sed -i "s/color/colour/g"
sed -i "s/color/colour/g" tex/ch1/intro.tex
sed -i "s/color/colour/g" tex/ch1/motivation.tex
....

The -n 1 argument will make xargs turn each line into a command of its own. The sed -i "s/color/colour/g" command will replace all occurrences of color with colour for the specified file.

Note that this only works if you don't have any spaces in your paths. If you do, you should use null terminated paths as input to xargs by passing the -0 flag. An example usage would be:

$ git ls-files -z "*.tex" | xargs -0 -n 1 sed -i "s/color/colour/g"

Which does the same as what we described above, but also works if one of the paths has a space in it.

This works with any command that produces filenames as output such as find or locate. If you do happen to use it in a git repository with a lot of files though, it might be more efficient to use it with git grep -l instead of git ls-files, like so:

$ git grep -l "color" "*.tex" | xargs -n 1 sed -i "s/color/colour/g"

The git grep -l "color" "*.tex" command will give a list of "*.tex" files containing the phrase "color".

Answer 4

Your first argument illustrates the difference quite well.

\ls | grep Cases | less lets you browse the list of file names produced by ls and grep. It doesn't matter that they happen to be file names, they're just some text.

\ls | grep Cases | xargs less lets you browse the files whose names are produced by the first part of the command. xargs takes a list of file names as input and a command on its command line, and runs the command with the file names on its command line.

When considering using xargs, keep in mind that it expects input formatted in a strange way: whitespace-delimited, with \, ' and " used for quoting (in an unusual way, because \ isn't special inside quotes). Only use xargs if you your file names don't contain whitespace or \'".

Answer 5

In your example you don't need to use xargs at all since find will do exactly and safely what you want to do.

Exactly what you want using find is:

find -maxdepth 1 -name '*Cases*' -exec touch {} +

In this example -maxdepth 1 means only search in the current directory, don't descend into any subdirectories; by default find will look in all subdirectories (which is often what you want) unless you constraint it with maxdepth. The {} is the name of the file that will get substituted in its place and the + is one of two end-of-command markers, the other being ;. The difference between them is that ; means exec the command on each file one at a time, whereas + means exec the command on all the files at once. Note, however, that your shell will probably try to interpret ; itself, so you will need to escape it with either \; or ';'. Yes, find has a number of little annoyances like this, but its power more than makes up for it.

Both find and xargs are tricky to learn at first. To help you learn xargs try using the -p or --interactive option which will show you the command it is about to execute and prompt you whether or not you want to run it.

Similarly with find you can use -ok in place of -exec to prompt you whether or not you want to run the command.

There are times, though, when find won't be able to do everything you want and that is where xargs comes in. The -exec command will only accept one instance of {} appearing, so if you would get an error with find -type f -exec cp {} {}.bak \; so you could instead do it like so: find -type f -print0 | xargs -0 -l1 -IX cp X X.bak

You can learn more about Run Commands in the GNU Findutils manual.

Also, I mentioned that find safely does what you want because when you are dealing with files you are going to encounter spaces and other characters that will cause problems with xargs unless you use the -0 or --null option along with something that generates input items terminated by a null character instead of whitespace.

Answer 6

xargs (along with find, sort, du, uniq, perl and a few others) accepts a command-line switch to say "STDIN has a list of files, separated by a NUL (0x00) byte". This makes it easy to handle filenames with spaces and other funny characters in them. Filenames don't contain NULs.