4

I've seen Why does 'find -exec cmd {} +' need to end in '{} +'? which explains why {} + has to be at the end of the -exec cmd(*), and now I'd like to know why GNU xargs reverts to one command per input argument when you use -I (or the deprecated -i)...or as the man page puts it

-I replace-str [...] Implies -x and -L 1.

For example:

$ seq 1 10 | xargs echo
1 2 3 4 5 6 7 8 9 10
$ seq 1 10 | xargs -I {} echo {}
1
2
3
4
5
6
7
8
9
10

The BUGS section of the xargs man page only says:

The -L option is incompatible with the -I option, but perhaps should not be.

and

When you use the -I option, each line read from the input is buffered internally. This means that there is an upper limit on the length of input line that xargs will accept when used with the -I option. To work around this limitation, you can use the -s option to increase the amount of buffer space that xargs uses, and you can also use an extra invocation of xargs to ensure that very long lines do not occur.

Neither of which explains why.

One of the main reasons to use xargs is to reduce the number of commands executed (running one cp or mv with 1000 source filenames is better than running 1000 cps or mvs with 1 source filename each) and this limitation prevents that in many cases....most cases since most unix commands want the source (e.g. a list of files) before the destination (e.g. a directory).

So, why does xargs have this limitation?


Update 2015-11-05

I recently created a freebsd 10.0 VM and found that freebsd's version of xargs has a -J option to deal with this issue.

-J replstr

If this option is specified, xargs will use the data read from standard input to replace the first occurrence of replstr instead of appending that data after all other arguments. This option will not affect how many arguments will be read from input (-n), or the size of the command(s) xargs will generate (-s). The option just moves where those arguments will be placed in the command(s) that are executed. The replstr must show up as a distinct argument to xargs. It will not be recognized if, for instance, it is in the middle of a quoted string. Furthermore, only the first occurrence of the replstr will be replaced. For example, the following command will copy the list of files and directories which start with an uppercase letter in the current directory to destdir:

/bin/ls -1d [A-Z]* | xargs -J % cp -Rp % destdir

(*) Essentially, "because the POSIX spec says so". IMO they should have tried harder to find a solution that allowed {} to appear anywhere in the -exec ... + , or at least required standard tools like cp and mv to have an option like GNU's -t option to reverse source and target.

2

(As this does not answer the question, this is should have been a comment, but is too long - so treat it as a comment).

As an alternative to FreeBSD's xargs you can use GNU Parallel which does not have this limitation. It even supports repeating the context:

seq 10 | parallel -Xj1 echo con{}text 
seq 10 | parallel -mj1 echo con{}text 

GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to. It can often replace a for loop.

If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:

Simple scheduling

GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:

GNU Parallel scheduling

Installation

If GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README

Learn more

See more examples: http://www.gnu.org/software/parallel/man.html

Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html

Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel

  • Doesn't really answer my question (and i already knew about parallel)....but +1 for a great answer, anyway. I occasionally use xargs -P, I should probably use this instead...i tried it a few times in the past but impatiently gave up on it when it wasn't obvious how to make it do what i wanted (i can't even remember what that was, it was years ago). – cas Jul 5 '16 at 11:26
-2

POSIX does not define things, POSIX just describes existing implementations to become the standard.

find -exec + was introduced in 1989 for SVr4 by David Korn and given the fact that it was added much later to the standard, there is few chance to change it.

But there is no problem with the current standard, as you could always call:

find . -type f -exec sh -c 'somecommand "$@" somedir' sh {} +

So you see there is no limitation and there is no need to use nonstandard options.

Regarding xargs: You are asking about internals of a vendor specific implementation. It is obvious that is is most improbable that you will be able to get a useful answer to your question here unless the author of the vendor specific implementation is reading and writing here.

My example however also works in a similar way for xargs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.