2

This:

#!/bin/bash

# Run command ~100Kbytes long
/bin/echo $(perl -e 'print "x"x100000') | wc
# Run command ~54Kbytes long
# This line fails: line 7: /bin/echo: Argument list too long
/bin/echo $(perl -e 'print "x "x27000') | wc

# Same command, but run using xargs
# Run command ~100Kbytes long
perl -e 'print "x"x100000' | xargs -n 100000 /bin/echo | wc
# Run command ~54Kbytes long
# This line fails: xargs: /bin/echo: Argument list too long
perl -e 'print "x "x27000' | xargs -n 100000 /bin/echo | wc

works fine on GNU/Linux, but the 2 54Kbytes lines fail on MacOS X.

ARG_MAX is way higher than 100KBytes, and please note the 100Kbytes lines does not fail - it is the 54KBytes line that fails.

mac$ getconf ARG_MAX
262144
mac$ uname -a
Darwin macosx 11.4.2 Darwin Kernel Version 11.4.2: Thu Aug 23 16:26:45 PDT 2012; root:xnu-1699.32.7~1/RELEASE_I386 i386
# Kusalananda suggests it may be due to the size of the environment
mac$ env | wc
      27      32     956

Why do the 54Kbytes command fail?

Is there a way I can predict if the argument list is too long for MacOS X without running it?

Research

This:

#!/bin/bash

runtest() {
    echo environment size:
    env | wc
    echo Run command ~100Kbytes long
    /bin/echo $(perl -e 'print "x"x100000') | wc
    echo Run command ~54Kbytes long
    # This line fails: line 7: /bin/echo: Argument list too long
    /bin/echo $(perl -e 'print "x "x27000') | wc
    
    # Same command, but run using xargs
    echo Run command ~100Kbytes long
    perl -e 'print "x"x100000' | xargs -n 100000 /bin/echo | wc
    echo Run command ~54Kbytes long
    # This line fails: xargs: /bin/echo: Argument list too long
    perl -e 'print "x "x27000' | xargs -n 100000 /bin/echo | wc
    echo
}

# Clean environment
runtest

# Make a huge environment
for a in `seq 5000`; do eval "a$a=1" ; done
for a in `seq 5000`; do eval "a$a() { 1; }" ; done
# This works as before
runtest

# Export environment
for a in `seq 5000`; do eval export a$a ; done
for a in `seq 5000`; do eval export -f a$a ; done
# Now the 100Kbytes commands fail, too
runtest

gives this output:

environment size:
    6027    6032   47849
Run command ~100Kbytes long
       1       1  100001
Run command ~54Kbytes long
test: line 10: /bin/echo: Argument list too long
       0       0       0
Run command ~100Kbytes long
       1       1  100001
Run command ~54Kbytes long
xargs: /bin/echo: Argument list too long
       0       0       0

environment size:
    6027    6032   47849
Run command ~100Kbytes long
       1       1  100001
Run command ~54Kbytes long
test: line 10: /bin/echo: Argument list too long
       0       0       0
Run command ~100Kbytes long
       1       1  100001
Run command ~54Kbytes long
xargs: /bin/echo: Argument list too long
       0       0       0

environment size:
   16027   26032  126742
Run command ~100Kbytes long
test: line 7: /bin/echo: Argument list too long
       0       0       0
Run command ~54Kbytes long
test: line 10: /bin/echo: Argument list too long
       0       0       0
Run command ~100Kbytes long
xargs: insufficient space for argument
       0       0       0
Run command ~54Kbytes long
xargs: /bin/echo: Argument list too long
       0       0       0

So Kusalananda is right that the exported environment can have an effect. It is unclear what the formula is to compute this: Maybe it is purely the size? Maybe the number of variables is important, too? Maybe it is only the lengths of the names? Maybe a linear combinations of those?

It still does not explain that in a given environment a 100Kbytes command works fine, but a 54Kbytes command does not.

It is as if MacOS not only has a limit on the total size but also has a limit on the number of args.

The numbers would also make sense if MacOS uses an additional 8 bytes per argument:

# One big argument
100K * "x" = 100000+2 < 262144 # Works
# 27K small arguments
27K * "x " = 27K*(8+2) > 262144 # Fails
# 26K small arguments
26K * "x " = 26K*(8+2) < 262144 # Works

But does MacOS do that?

10
  • Run the command again in an empty environment. The size of the environment also matters, i.e. the length of each environment variable and their values must also fit within the ARG_MAX size.
    – Kusalananda
    Aug 13, 2020 at 6:19
  • @Kusalananda Good call. I need to be able to predict if a command can run in an unclean environment. How can I compute this?
    – Ole Tange
    Aug 13, 2020 at 6:30
  • @Kusalananda Also: Why does the big command run, when the small command does not? How does the environment affect this?
    – Ole Tange
    Aug 13, 2020 at 6:33
  • It's to do with the number of arguments. Note that using xx in place of x (followed by a space character) makes it work as it produces a single large argument rather than 27k small ones.
    – Kusalananda
    Aug 13, 2020 at 6:45
  • @Kusalananda I reckoned as much. But where is the 27K limit set? How can I get MacOS to tell me that? Having worked with UNIX for > 20 years I have never seen a command line that was limited by number of args - only by the total length.
    – Ole Tange
    Aug 13, 2020 at 6:54

1 Answer 1

1

Further research indicates (MacOS version unknown):

Effective length = 
  length of arguments +
  5 * number of arguments +
  length of body/value of exported functions/variables +
  length of names of exported functions/variables +
  4 * number of exported functions/variables

If effective length < 256 Kbytes the command will run. It is unclear if this is true for all versions of MacOS.

For MacOS El Capitan 10.11.4 this gives the pessimistic command line length (assuming the command you want to run is is /bin/echo x x x x ...):

perl -e '                                                                             
  $envc=(keys %ENV);                                                                    
  $envn=length join"",(keys %ENV);                                                      
  $envv=length join"",(values %ENV);                                                    
  $maxlen=3+(262144 - $envn - $envv) / 5 - $envc*2;                                     
  print("Max len = $maxlen\n");
'                                                        

It is unclear if this is true for all versions of MacOS.

2
  • In your length of arguments (argv[]), do you include the '\0' delimiters? Same for environment strings (envp[]), they are of the format var=value\0. So I'd expect 2 bytes on top of the length of the name and value. In anycase, exported functions/variables are a concept of the shell (some shells), only relevant here in that those end-up being encoded in envp[] one way or another. But more things can and do end up in envp[] like the _=/path/to/executable, or the env vars that shells (some shells) received on startup and cannot be mapped to shell variables. Aug 17, 2020 at 13:33
  • @StéphaneChazelas The lengths are measured using wc. But \0 could be one of the 5, which would then be 1+4.
    – Ole Tange
    Aug 17, 2020 at 21:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .