13

This question already has an answer here:

Context for the question: According to POSIX specs, ARG_MAX is maximum length of command-line arguments to exec() family of functions. Which lead me to believe that's actual number of arguments, however that clearly didn't work:

$ ulimit -s
8192
$ touch {1..18000}.jpg
$ rm *.jpg
$ 

Clearly, this works fine, despite being in length over 8192 items. According to D.W.'s answer, the 8192 is supposedly size in kB. So clearly the previous assumption was wrong.

This is where the actual question comes in: How do I figure out the amount of items that actually will get above 8192 kB limit ? In other words, what sort of computation I have to perform to ensure that *.jpg type of glob will result into Argument list too long error ?

Please note, this isn't a duplicate of What defines the maximum size of single command argument. I know about getconf ARG_MAX and ulimit -s values, that's not my question. I need to know how to generate enough arguments in size that will be above the limit. In other words, I need to find a way to get the error, not avoid it.

marked as duplicate by Jesse_b, schily, Rui F Ribeiro, Romeo Ninov, Kiwy Jun 25 '18 at 11:59

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • What is the result of getconf ARG_MAX for you? – Jesse_b Jun 23 '18 at 18:03
  • @Jesse_b 2097152 . And I'm on Linux, btw – Sergiy Kolodyazhnyy Jun 23 '18 at 18:06
  • Are you trying to manually reproduce xargs? – Jeff Schaller Jun 23 '18 at 18:23
  • @JeffSchaller No, I wanted to generate enough files to play around with various methods of handling Argument list too long error. So far I've figured out ulimit -s gives stack space, ARG_MAX is subset to that. I see so far that getconf ARG_MAX gives basically 2M by default for argument size, 8M for stack space. So basically all I need to know is how to create enough filenames to go beyond 2M limit, so that when I do *.jpg in any command that involves exec() family of function would throw that error. – Sergiy Kolodyazhnyy Jun 23 '18 at 18:33
  • 1
    Looks useful: unix.stackexchange.com/a/110301/117549 – Jeff Schaller Jun 23 '18 at 19:11
9

Using getconf ARG_MAX to generate a long list of x and calling an external utility with that as its argument would generate an "Argument list too long" error:

$ /bin/echo $( perl -e 'print "x" x $ARGV[0]' "$(getconf ARG_MAX)" )
/bin/sh: /bin/echo: Argument list too long

The environment and the length of the string /bin/echo will be included in what makes the error occur, so we can try to find the biggest possible number by subtracting these:

$ env
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/X11R6/bin:/usr/local/bin

(I started this shell with env -i sh, so there's only the PATH variable in the environment)

$ /bin/echo $( perl -e 'print "x" x ($ARGV[0] - length($ENV{"PATH"}) - length("/bin/echo"))' "$(getconf ARG_MAX)" )
sh: /bin/echo: Argument list too long

Still too long. By how much?

i=0
while ! /bin/echo $( perl -e 'print "x" x ($ARGV[0] - length($ENV{"PATH"}) - length("/bin/echo") - $ARGV[1])' "$(getconf ARG_MAX)" "$i" )
do
    i=$(( i + 1 ))
done

This loop exits for i=8.

So there's four bytes that I can't immediately account for (four of the eight must be for the name of the PATH environment variable). These are the null terminators for the four strings PATH, the value of PATH, /bin/echo and the long string of x characters.

Note that each argument is null terminated, so the more arguments you have to the command, the shorter the combined length of them can be.


Also, just to show the effect of a big environment:

$ export BIG=$( perl -e 'print "x" x $ARGV[0]' "$( getconf ARG_MAX )" )

$ /bin/echo hello
sh: /bin/echo: Argument list too long

$ /bin/echo
sh: /bin/echo: Argument list too long
  • So ARG_MAX is actually a character count for the whole command line, or is it rather a buffer size? How does character encoding play into this? – dessert Jun 24 '18 at 8:21
  • 2
    @dessert ARG_MAX is the number of bytes that the combined length of the environment and the command argument list, including the command name and any null terminators on each argument (and on environment variable names/values), can be. So multi-byte characters would take more space. It may be a kernel or library buffer size. – Kusalananda Jun 24 '18 at 8:28
  • @dessert Unquoted whitespace between the command's arguments would not count, so it's not the length of the command line. – Kusalananda Jun 24 '18 at 8:29
  • 2
    @Kusalananda, I've understood it's the limit on the argument string size, it's discussed in the linked question. E.g. this would trigger it: /bin/echo $( perl -e 'print "x" x $ARGV[0]' 131072 ) > /dev/null (one less works). I think PATH_MAX is usually 4096 (again, on Linux), and I don't think it's related. – ilkkachu Jun 24 '18 at 20:14
  • 1
    @ilkkachu Thanks for pointing this out. That limit does not seem to be an issue on OpenBSD, but I'll find a Linux system somewhere and may update the answer when I have time. – Kusalananda Jun 24 '18 at 20:23
9

On many Linux distributions, you can find out what's the current value of ARG_MAX by running

getconf ARG_MAX

A simple way to generate the message "Argument list too long" is to actually supply a long argument list; for example, on my system

/bin/echo "$(find / -xdev 2> /dev/null)"

works, but

/bin/echo "$(find / -xdev 2> /dev/null)" "$(find / -xdev 2> /dev/null)"

produces

bash: /bin/echo: Argument list too long

For an in-depth discussion, see "Argument list too long error for rm, cp, mv commands" on Stack Overflow.

P.S. The length in question is the actual amount of memory needed to store the arguments and environment. It is not related to the number of arguments.

  • It's ever so slightly related to the number of arguments since each argument would need a null terminator. So having helloworld would use 11 bytes, while hello and world would use 12. – Kusalananda Jun 24 '18 at 8:36
6

Instead of listing all the files on the system, which takes forever and doesn't work on sparse systems, the command

/bin/echo {1..200000}

is a much faster (163ms) way to generate the error. For me ARG_MAX is 2097152 (2,097,152 on the order of 2 million) but the command still errors.

If your shell doesn't have {start..end}, you can use the somewhat slower (but still faster than listing files)

/bin/echo $(seq 1 200000)

If either command doesn't work, just increase the end value until it does, until bash runs out of memory, or until seq can't count any higher.

  • You don’t need seq for this, /bin/echo {1..200000} also works. – Lucas Werkmeister Jun 24 '18 at 9:00
  • @LucasWerkmeister: Only if the shell supports this feature. – David Foerster Jun 24 '18 at 9:27
  • @DavidFoerster sure, but you mentioned Bash in your comment already. – Lucas Werkmeister Jun 24 '18 at 13:32
  • @LucasWerkmeister good point, it's also exponentially faster – cat Jun 24 '18 at 18:19
6

To make it easier start a new shell (exit when done) and set the limit to a lower value (100kbytes):

$ bash
$ ulimit -s 100

With the example you used, the error could be triggered:

$ touch {1..100000}.jpg
bash: /usr/bin/touch: Argument list too long

But better to use something like echo. Understand that a builtin might not trigger an error, this command should work correctly:

$ echo {000001..100000}6789

Also note that the amount of bytes is well over the limit set above (1.1Mbyte):

$ echo {000001..100000}6789 | wc -c
1100000

But this command won't work:

$ /bin/echo {000001..100000}6789
bash: /bin/echo: Argument list too long

The limit is set by the size of the argument list in bytes added to the environment size of the shell.

Not the answer you're looking for? Browse other questions tagged or ask your own question.