2

Why are the numbers so irregular?

echo {1..200000} | xargs perl -E 'say "ok:", scalar @ARGV'
ok:23691
ok:21840
ok:21840
ok:21840
ok:20261
ok:18720
ok:18720
ok:18720
ok:18720
ok:15648

It's more civilised with standard argument length.

perl -E' say "1 " x 900000' | xargs perl -E 'say "ok:", scalar @ARGV'
ok:65520
ok:65520
ok:65520
ok:65520
ok:65520
ok:65520
ok:65520
ok:65520
ok:65520
ok:65520
ok:65520
ok:65520
ok:65520
ok:48240

What's the key factor anyhow?

3
  • 2
    "so irregular": It is decreasing Jul 29, 2021 at 6:15
  • @rexkogitans And they are all integers!
    – user422875
    Jul 29, 2021 at 10:47
  • 1
    I meant it seriously, because the keypoint is: Since the single arguments get longer, fewer arguments fit into the same maximum total length. Jul 29, 2021 at 12:09

3 Answers 3

5

The numbers that matter are the total length of (all) the arguments, and the command buffer size xargs decides to use.

The first depends on the fixed command line to the command you run, and the arguments xargs gives each invocation. perl -E 'say "ok:", scalar @ARGV' is 32 bytes, counting the NUL bytes that terminate the strings (i.e. perl<NUL>-E<NUL>say "ok:", scalar @ARGV<NUL>. And in the second example, all the arguments are two bytes each, 1<NUL>. So 32 + 65520 * 2 bytes, or 131072 B = 128 * 1024 B = 128 kB.

Obviously in the first example, the lengths of arguments vary, giving varying counts, but the logic should be the same. E.g. 21840 args for the second to fourth runs matches 5-digit arguments (6 bytes each): 21840 * 6 + 32 = 131072.

The size of the command buffer may depend on the implementation, but GNU xargs can show it with xargs --show-limits, and on my Linux, I get:

$ echo | xargs --show-limits
Your environment variables take up 2305 bytes
POSIX upper limit on argument length (this system): 2092799
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2090494
Size of command buffer we are actually using: 131072
Maximum parallelism (--max-procs must be no greater): 2147483647

Looking at the second to last line, that's exactly the same number.

You can change the size of the buffer it uses with -s, e.g. with just 10 kB buffer:

$ perl -E' say "1 " x 90000' | xargs -s 10240 perl -E 'say "ok:", scalar @ARGV' 
ok:5104
ok:5104
ok:5104
...

And of course there's also -n to limit the number of individual arguments:

$ echo {1..200000} | xargs -n 10000 perl -E 'say "ok:", scalar @ARGV' 
ok:10000
ok:10000
ok:10000
...

--show-limits mentions environment variables because they use the same space as command line arguments, and if you raise the buffer size enough, close to the system maximum, their size starts to matter too.

I'm not sure if the system also counts the sizes of the pointers to the argument strings against the limit, but at least xargs doesn't seem to care about that.

3

For background context: Linux and other Unices have a limit on the maximum number of arguments which can be passed to a command, and in some cases the total length of the resultant command line. xargs needs to deal with this, as otherwise when it goes to execute commands with a huge number of arguments (like in this case) it would simply get E2BIG at execvp time and fail.

Many resources online will tell you that ARG_MAX is the canonical limit for arguments, but they're mistaken: ARG_MAX is for a single argument, and even then, there are many more limits that may take effect before this in practice. These limits just as often operate based on the number of bytes in the resultant command line, rather than the number of arguments (ie. pointer array based). In this case, you're brushing up against the fact that, in the GNU version of xargs, the point at which to split commands is determined in userspace based on a fixed length (bc_args_exceed_testing_limit). You can see some of these limits with xargs --show-limits if you're using GNU xargs:

% xargs --show-limits </dev/null |& grep -e arg -e command
POSIX upper limit on argument length (this system): 2090868
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2086632
Size of command buffer we are actually using: 131072

If you'd prefer to see these limits at the source, in xargs/xargs.c, we can see that we fork() off the child, and then do a test for argument length prior to exec() in it:

[...]

      /* If we run out of processes, wait for a child to return and
         try again.  */
      while ((child = fork ()) < 0 && errno == EAGAIN && procs_executing)
        wait_for_proc (false, 1u);

      switch (child)
        {
        case -1:
          die (EXIT_FAILURE, errno, _("cannot fork"));

        case 0:                /* Child.  */
          {
            close (fd[0]);
            child_error = EXIT_SUCCESS;

            prep_child_for_exec ();

            if (bc_args_exceed_testing_limit (argv))
              errno = E2BIG;
            else
              execvp (argv[0], argv);

[...]

This check comes from lib/buildcmd.c:

/* Return nonzero if the indicated argument list exceeds a testing limit.
 * NOTE: argv could be declared 'const char *const *argv', but it works as
 * expected only with C++ compilers <http://c-faq.com/ansi/constmismatch.html>.
 */
bool
bc_args_exceed_testing_limit (char **argv)
{
  size_t chars, args;

  for (chars=args=0; *argv; ++argv)
    {
      ++args;
      chars += strlen(*argv);
    }

  return (exceeds ("__GNU_FINDUTILS_EXEC_ARG_COUNT_LIMIT", args) ||
          exceeds ("__GNU_FINDUTILS_EXEC_ARG_LENGTH_LIMIT", chars));
}

Since your later numbers are longer in string form, they take more space, and thus result in new commands more eagerly by xargs.

2

xargs needs to fit the arguments into a set size that is limited by your OS - the first example has arguments that grow, so less fit per call to perl. You can fit more 3-character arguments than 5-character ones into the limit.

The second is always "1" as the argument, so 65520 of those fit every time.

You must log in to answer this question.