0

xargs has an option -n for specifying the maximum number of arguments to a command to run at each time.

Is there a way to specify that xargs should always run any command with all the provided arguments at once? (Just like running the command with all the arguments directly without xargs, I am not trying to avoid failure due to system limit on command line length)

The following is not my question. Please disregard it, if it distracts you. I don't want to use command substitution, because it removes NUL and trailing newlines, so I am considering to use xargs as an alternative. But I don't want xargs to group the args for several runs, because the results from multiple runs with different subsets of arguments can be different from results from single run with all the arguments, depending on the command that xargs executes. So I want to tell xargs to always run all the arguments at once.

  • 2
    One of tho reasons for using xargs is that very long lists of arguments simply can't be given to a utility in one go as the argument list has an upper length limit. xargs will by default give as many arguments as possible to the utility. An example of your issue would be good to see (please edit your question). – Kusalananda Nov 23 '18 at 13:07
  • If you want to pass chunks of data containing NULs via the command line (via the arguments to execve(2)), forget about it. This is not possible, in any language. If you want to bypass the length limit imposed by the OS on the args + env and run a prog with a huge nr of args, there may be hacky ways of doing it, but not with the standard tools. – mosvy Nov 23 '18 at 13:40
  • Why is it not possible If you want to pass chunks of data containing NULs via the command line (via the arguments to execve(2))?@mosvy – Ben Nov 23 '18 at 13:53
  • Because NUL terminates individual arguments in argv. – Stephen Kitt Nov 23 '18 at 13:54
  • Could you explain what you’re trying to do in a bit more detail? I imagine you’ve got either a file, or a command outputting some data, which you want to use as arguments for another command, but only run the latter command if you can do so with all the data as arguments — is that correct? Can the second command take its input from its standard input instead? – Stephen Kitt Nov 23 '18 at 14:09
1

I'm still not sure I understand what you're after, but combining the -x option ("exit if the command line doesn't fit") with the -n option set to a huge value (larger than the system limit) should:

a) make sure xargs is run just once, no matter how many arguments are given to it

b) error out if the arguments could not be fit in a single command because of the OS or xargs-internal limit.

Example:

$ seq 1 10000 | xargs -n 100000000 -x sh -c 'echo "$#"' sh
10000
$ seq 1 100000 | xargs -n 100000000 -x sh -c 'echo "$#"' sh
xargs: argument list too long

Unfortunately, this does not work with the BSD or solaris' xargs. On *BSD, the -x option will cause xargs to run its command with a single argument, instead of exiting:

fz11_2$ jot 10000 1 | xargs -n 10000 -x sh -c 'echo $#' sh | head -3
1
1
1
xargs: sh: terminated with signal 13; aborting

Only some ridiculously small argument to -s will cause the -x to trigger:

fz11_2$ jot 10000 1 | xargs -s 19 -n 10000 -x sh -c 'echo $#' sh | head -3
xargs: insufficient space for arguments

The standard seems to match the GNU xargs behavior:

-n  number
      Invoke utility using as many standard input arguments as
      possible, up to number (a positive decimal integer) arguments
      maximum. Fewer arguments shall be used if:
      + The command line length accumulated exceeds the size specified
        by the -s option (or {LINE_MAX} if there is no -s option).
      + The last iteration has fewer than number, but not zero,
        operands remaining.
-x
      Terminate if a constructed command line will not fit in the
      implied or specified size (see the -s option above).
2

What you describe is already the default. xargs -n provides a maximum number of arguments to use, but if you don't specify it, xargs will use as many arguments as possible.

You mentioned the system limit in your question, and that you don't expect/need to circumvent that. xargs will take that system limit into account, but may use a lower value, as described in the usage for the --max-chars (-s) option:

 --max-chars=max-chars
 -s max-chars

Use at most max-chars characters per command line, including the command and initial-arguments and the terminating nulls at the ends of the argument strings. The largest allowed value is system-dependent, and is calculated as the argument length limit for exec, less the size of your environment, less 2048 bytes of headroom. If this value is more than 128KiB, 128Kib is used as the default value; otherwise, the default value is the maximum. 1KiB is 1024 bytes.

(emphasis mine)

You can check what it is with echo | xargs --show-limits. Example output:

Your environment variables take up 3712 bytes
POSIX upper limit on argument length (this system): 2091392
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2087680
Size of command buffer we are actually using: 131072
Maximum parallelism (--max-procs must be no greater): 2147483647

This shows that my hard limit is 2087680 (just under 2MiB). If your system also permits more than 128K, you can prevent splitting by using xargs -s 2087680 (adjust according to your own limit).

Older versions of xargs would even let you provide a higher limit than the OS permits. With 4.4.2, a warning was displayed, but the specified value accepted anyway, and you get the same error that you would see from running the command directly:

$ seq 1 2000000 | xargs -s 2100000 echo | wc -l
xargs: value for -s option should be < 2092927
xargs: echo: Argument list too long
0

But in versions 4.6, the hard limit is respected:

$ seq 1 2000000 | xargs -s 2100000 echo | wc -l
xargs: value for -s option should be <= 2091392
23
  • I would like xargs to always run any command with all the arguments given at once. – Ben Nov 23 '18 at 14:33
  • Please could you show a situation in which it doesn't? – JigglyNaga Nov 23 '18 at 14:34
  • That is what my post is asking for. – Ben Nov 23 '18 at 14:36
  • 1
    No, your post doesn't show the situation where xargs is unsuitable. It will only split if either -n is provided (which you're not doing) or the --max-chars/system limit is exceeded (which you've implied you're not reaching). – JigglyNaga Nov 23 '18 at 14:38
  • I was asking "Is there a way to specify that xargs should always run any command with all the provided arguments at once? (Just like running the command with all the arguments directly without xargs, I am not trying to avoid failure due to system limit on command line length)". The system limit can be reached, and I am looking for some way which makes xargs fail. – Ben Nov 23 '18 at 14:41
2

Some clarifications:

In

cmd1 $(cmd2)

Here is what happens with the output of cmd2:

  • in bash, the NULs are discarded. YMMV with other shells (in zsh they're preserved, some shells ignore everything after the first NUL).
  • the trailing newline characters are removed
  • the result is split according to $IFS whose default value is SPC, TAB, NL in most shells, SPC, TAB, NL, NUL in zsh.
  • each work is subject to globbing (except in zsh). For instance * becomes the list of non-hidden files in the current directory.
  • the resulting list is passed as arguments to cmd1. The execve() system call to do that will fail if that list (combined with the environment) is too big (and in the case of Linux also if any single argument is larger than 128KiB).

In

cmd2 | xargs cmd1

Here's what happening with the output of cmd2:

  • in some implementations, the bytes of that output are interpreted as characters. For instance in a locale using a UTF-8 charset the e2 80 86 byte sequence makes up a U+2006 character which is the SIX-PER-EM SPACE character, often considered as a blanks character. If some bytes don't form valid characters, depending on the implementation, they may be removed or cause a failure, the behaviour is not specified by POSIX if the input is not text
  • If there are NUL bytes, again, that's not text, so the behaviour varies with the implementation
  • If there are lines that are bigger than LINE_MAX (which on some systems can be as low as 1024), the behaviour is also unspecified.
  • xargs parses its input as per its own expected format. In that format, whitespace and newline characters are understood as delimiter (so trailing newlines would be split all the same) the list of which varies with the implementation (including that U+2006 for instance in some), and '...', "...", and backslash are used as quoting operators to prevent those delimiters and other quoting operators to be treated specially.
  • that results in a list or words.
  • in some implementations, if any of those words is _, it's treated as an end-of-file logical indicator and everything after it is ignored.
  • if any of those words is larger than some limit (in many systems, as low as 255 bytes), the behaviour is unspecified.
  • the remaining words are passed as arguments to cmd1, but if that list is too big, instead of failing like in the command substitution case, xargs tries to fit as many as possible (with a bit of margin which varies with the implementation) for execve() to succeed, and runs the command as many times as needed to use all the arguments.

Now:

  • if you wanted to pass the whole output of cmd2 as one argument to cmd1:
    • if that output contains NUL characters, you can't as that's a limitation of the execve() system call, not command substitution nor xargs.
    • if that output is bigger than the ARG_MAX limit, you can't (though there are ways to increase that limit on some systems), again a limitation of exexve(). You could if cmd1 was for instance a builtin or function of your shell (as there's no execve() involved then), but even then, in practice, only zsh supports arguments with NUL in that case.
    • on Linux, if it's bigger than 128KiB, you can't. That limit can't be raised.
    • Otherwise (if there's no NUL, and it's small enough), you can always do: out=$(cmd2; echo .); cmd1 "${out%.}" or with GNU xargs: cmd2 | xargs -0 cmd1
  • "If there are NUL bytes, again, that's not text, so the behaviour varies with the implementation". Do you mean the behaviour of xargs without -0? I guess xargs with -0 expects NUL bytes? – Ben Nov 24 '18 at 16:50
  • @Ben, yes -0 is a GNU extension, out of scope of the POSIX specification anyway. -0 is pretty simple and without surprises in all the implementations that support it and doesn't interpret the input as text, just sequences of bytes without any interpretation. GNU xargs used to be very buggy and unmaintained until the mid 2000s. The only thing you could rely on with it was -0 back then. – Stéphane Chazelas Nov 24 '18 at 16:56
  • What is GNU xargs' behavior If there is any line bigger than LINE_MAX? (note that in GNU findutils' manual, LINE_MAX is not mentioned, and I guess it probably is named something else?) Is LINE_MAX the one overridded by -s? – Ben Nov 24 '18 at 17:11
  • "if any of those words is larger than some limit (in many systems, as low as 255 bytes), the behaviour is unspecified." Is the limit on a word length specified by some variable in Linux and in GNU xargs? – Ben Nov 24 '18 at 17:15
  • @Ben, GNU utilities generally don't have max line limits. Traditional Unix versions do as they tend to use plain stdio (fgets() and co) which also explains why they can't cope with NULs in input lines. – Stéphane Chazelas Nov 24 '18 at 17:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.