Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

The xargs command always confuses me. Is there a general rule that can help me figure out when I need it?

Consider the two examples below:

$ \ls | grep Cases | less

prints the files that match 'Cases', but changing the command to touch will require xargs:

$ \ls | grep Cases | touch
touch: missing file operand
Try `touch --help' for more information.

$ \ls | grep Cases | xargs touch
share|improve this question

6 Answers 6

up vote 59 down vote accepted

The difference is in what data the target program is accepting.

If you just use a pipe, it receives data on STDIN (the standard input stream) as a raw pile of data that it can sort through one line at a time. However some programs don't accept their commands on standard in, they expect it to be spelled out in the arguments to the command. For example touch takes a file name as a parameter on the command line like so: touch file1.txt.

If you have a program that outputs filenames on standard out and want to use them as arguments to touch, you have to use xargs which reads the STDIN stream data and converts each line into space separated arguments to the command.

These two things are equivalent:

# touch file1.txt
# echo file1.txt | xargs touch

Don't use xargs unless you know exactly what it's doing and why it's needed. It's quite often the case that there is a better way to do the job than using xargs to force the conversion. The conversion process is also fraught with potential pitfalls like escaping and word expansion etc.

share|improve this answer
    
Thanks for the caveat, wasn't aware of the pitfalls. –  Zaid Nov 19 '11 at 16:52
2  
The warning feels a little string to me. Of the two common options to get a stream onto a command line (xargs and $(...)), xargs is far safer than command substitution. And I cannot recall ever coming across a legitimate filename with a newline in it. Aren't the escaping and word expansion pitfalls issues with command substitution, not xargs? –  camh Nov 19 '11 at 22:31
5  
@camh: They're potential pitfalls with both. In the shell, you have to worry about filenames getting split on spaces, tabs, and newlines. In xargs, you only have to worry about newlines. In xargs, if you're output is formatted properly, you can split words/filenames on the NUL character instead (xargs -0), which is useful in conjunction with find -print0. –  Ken Bloom Nov 20 '11 at 1:22
    
Does xargs call the program via the shell with space separated args, or does it actually construct the argument list internally (eg. for use with execv/execp)? –  detly Nov 20 '11 at 9:55
1  
It constructs it internally and uses execvp, so it's safe. Also, GNU xargs (as used on Linux and a few others) lets you specify newline as your delimiter with -d \n, although BSD xargs (OSX et al) does not appear to support this option. –  fluffy Nov 21 '11 at 0:20

xargs is particularly useful when you have a list of filepaths on stdin and want to do something with them. For example:

$ git ls-files "*.tex" | xargs -n 1 sed -i "s/color/colour/g"

Let's examine this step by step:

$ git ls-files "*.tex"
tex/ch1/intro.tex
tex/ch1/motivation.tex
....

In other words, our input is a list of paths that we want to do something to.

To find out what xargs does with these paths, a nice trick is to add echo before your command, like so:

$ git ls-files "*.tex" | xargs -n 1 echo sed -i "s/color/colour/g"
sed -i "s/color/colour/g" tex/ch1/intro.tex
sed -i "s/color/colour/g" tex/ch1/motivation.tex
....

The -n 1 argument will make xargs turn each line into a command of its own. The sed -i "s/color/colour/g" command will replace all occurrences of color with colour for the specified file.

Note that this only works if you don't have any spaces in your paths. If you do, you should use null terminated paths as input to xargs by passing the -0 flag. An example usage would be:

$ git ls-files -z "*.tex" | xargs -0 -n 1 sed -i "s/color/colour/g"

Which does the same as what we described above, but also works if one of the paths has a space in it.

This works with any command that produces filenames as output such as find or locate. If you do happen to use it in a git repository with a lot of files though, it might be more efficient to use it with git grep -l instead of git ls-files, like so:

$ git grep -l "color" "*.tex" | xargs -n 1 sed -i "s/color/colour/g"

The git grep -l "color" "*.tex" command will give a list of "*.tex" files containing the phrase "color".

share|improve this answer

To expand on the answers already provided, some xargs implementations (like GNU or modern BSDs) can do one cool thing that is becoming increasingly important in today's multicore and distributed computing landscape: it can parallel process jobs.

For example:

$ find . -type f -name '*.wav' -print0 |xargs -0 -P 3 -n 1 flac -V8

will encode *.wav => *.flac, using three processes at once (-P 3).

share|improve this answer
1  
Cool use there! –  Zaid Nov 19 '11 at 21:26
    
Wow. I should have known this a week ago when I was doing exactly the same thing (except using OGG) with 50GiB of WAVs. :) –  Alois Mahdal May 13 '12 at 21:38
    
why not use the -exec parameter that find has? –  Evgeny Sep 11 '12 at 9:46
1  
@Evgeny The -exec parameter won't parallel-process jobs. –  amphetamachine Sep 18 '12 at 12:56

xargs (along with find, sort, du, uniq, perl and a few others) accepts a command-line switch to say "STDIN has a list of files, separated by a NUL (0x00) byte". This makes it easy to handle filenames with spaces and other funny characters in them. Filenames don't contain NULs.

share|improve this answer
1  
I think you mean "filenames can't contain nulls." –  amphetamachine Nov 23 '11 at 5:50

Your first argument illustrates the difference quite well.

\ls | grep Cases | less lets you browse the list of file names produced by ls and grep. It doesn't matter that they happen to be file names, they're just some text.

\ls | grep Cases | xargs less lets you browse the files whose names are produced by the first part of the command. xargs takes a list of file names as input and a command on its command line, and runs the command with the file names on its command line.

When considering using xargs, keep in mind that it expects input formatted in a strange way: whitespace-delimited, with \, ' and " used for quoting (in an unusual way, because \ isn't special inside quotes). Only use xargs if you your file names don't contain whitespace or \'".

share|improve this answer
    
@Gilles: xargs has the -0, --null option to get around the spaces issue (it's highly likely I learnt that from you :), so I assume that you are referring to a no-options xarg call, but I'm puzzled by your reference to the quotes. Do you have a link or an example regarding that? .. (ps. | xargs less is a handy "trick" +1.. thanks.. –  Peter.O Dec 1 '11 at 20:59

In your example you don't need to use xargs at all since find will do exactly and safely what you want to do.

Exactly what you want using find is:

find -maxdepth 1 -name '*Cases*' -exec touch {} +

In this example -maxdepth 1 means only search in the current directory, don't descend into any subdirectories; by default find will look in all subdirectories (which is often what you want) unless you constraint it with maxdepth. The {} is the name of the file that will get substituted in its place and the + is one of two end-of-command markers, the other being ;. The difference between them is that ; means exec the command on each file one at a time, whereas + means exec the command on all the files at once. Note, however, that your shell will probably try to interpret ; itself, so you will need to escape it with either \; or ';'. Yes, find has a number of little annoyances like this, but its power more than makes up for it.

Both find and xargs are tricky to learn at first. To help you learn xargs try using the -p or --interactive option which will show you the command it is about to execute and prompt you whether or not you want to run it.

Similarly with find you can use -ok in place of -exec to prompt you whether or not you want to run the command.

There are times, though, when find won't be able to do everything you want and that is where xargs comes in. The -exec command will only accept one instance of {} appearing, so if you would get an error with find -type f -exec cp {} {}.bak \; so you could instead do it like so: find -type f -print0 | xargs -0 -l1 -IX cp X X.bak

You can learn more about Run Commands in the GNU Findutils manual.

Also, I mentioned that find safely does what you want because when you are dealing with files you are going to encounter spaces and other characters that will cause problems with xargs unless you use the -0 or --null option along with something that generates input items terminated by a null character instead of whitespace.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.