5

I am trying to automate a process which involves running scripts on various machines via ssh. It is vital to capture both output and the return code (for the detection of errors).

Setting the exit code explicitly works as expected:

~$ ssh host exit 5 && echo OK || echo FAIL
FAIL

However, if there is a shell script signalling an unclean exit, ssh always returns 0 (script simulated by string execution):

~$ ssh host sh -c 'exit 5' && echo OK || echo FAIL
OK

Running the very same script on the host in an interactive shell works just fine:

~$ sh -c 'exit 5' && echo OK || echo FAIL
FAIL

I am confused as to why this happens. How can I tell ssh to propagate bash's return code? I may not change the remote scripts.

I am using public key authentication, the private key is unlocked – there is no need for user interaction. All systems are Ubuntu 18.04. Application versions are:

  • OpenSSH_7.6p1 Ubuntu-4ubuntu0.1, OpenSSL 1.0.2n 7 Dec 2017
  • GNU bash, Version 4.4.19(1)-release (x86_64-pc-linux-gnu)

Note: This question is different from these seemingly similar questions:

  • Are you certain that "sh" on the remote system is bash? – Jeff Schaller Dec 18 '18 at 19:04
  • 1
    It seems like there was a similar problem earlier, where the shell doesn't fork because there's a simple command. Does the behavior change if you ask it to run sh -c 'sleep 0.1; exit 5'? – Jeff Schaller Dec 18 '18 at 19:05
  • @JeffSchaller No, I am not certain. On Ubuntu, /bin/sh actually points to /bin/dash. Nevertheless the behaviour does not change if I use absolute paths (/bin/bash and /bin/sh instead of sh). I hope there is no further auto-redirect going on. – Hermann Dec 18 '18 at 19:41
  • @JeffSchaller Yes, even ssh host sh -c ':; exit 5' yields the expected return code. This does not help me as in the real world script, there is a lot more going on. I want to examine what forks where and then improve the examples in my question. – Hermann Dec 18 '18 at 19:43
7

I am able to duplicate this using the command you used, and I am able to resolve it by wrapping the remote command in quotes. Here are my test cases:

#!/bin/bash -x

echo 'Unquoted Test:'
ssh evil sh -x -c exit 5 && echo OK || echo FAIL

echo 'Quoted Test 1:'
ssh evil sh -x -c 'exit 5' && echo OK || echo FAIL

echo 'Quoted Test 2:'
ssh evil 'sh -x -c "exit 5"' && echo OK || echo FAIL

Here are the results:

bash-[540]$ bash -x test.sh
+ echo 'Unquoted Test:'
Unquoted Test:
+ ssh evil sh -x -c exit 5
+ exit
+ echo OK
OK
+ echo 'Quoted Test 1:'
Quoted Test 1:
+ ssh evil sh -x -c 'exit 5'
+ exit
+ echo OK
OK
+ echo 'Quoted Test 2:'
Quoted Test 2:
+ ssh evil 'sh -x -c "exit 5"'
+ exit 5
+ echo FAIL
FAIL

In the first test and second tests, it seems the 5 is not being passed to exit as we would expect it to be. It just seems to be disappearing. It's not going to exit, sh isn't complaining about 5: command not found, and ssh isn't complaining about it.

In the third test, exit 5 is quoted within the larger command to run on the remote host, same as in the second test. This ensures that the 5 is passed to exit, and both are executed as the -c option to sh. The difference between the second and third tests is that the whole set of commands and arguments is sent to the remote host quoted as a single command argument to ssh.

  • It does not help me in actually solving the problem I encounter, but I am accepting this answer as it shortly explains what is going on. – Hermann Dec 19 '18 at 13:59
  • Yeah. I would really love to know there the 5 is going in the first two tests. Especially the second where it's quoted to one level. I guess the remote shell is eating those quotes before it gets to sh. But I still don't get where the 5 is going. If I run sh -c echo 5, the 5 is nowhere to be seen, either. But sh -c 'echo 5' does what you expect and echoes 5. – Tim Kennedy Dec 19 '18 at 14:47
6

As noted in the answer you already have, the remote sh is not executing exit 5. Just exit:

$ ssh test sh -x -c 'exit 5'; echo $?
+ exit
0

What is happening here is explained, for instance, in this answer:

ssh executes a remote shell and passes a string to it, not a list of arguments.

When we execute ssh host sh -c 'exit 5':

  1. The local shell removes the single quotes (quote removal);
  2. The ssh client gets the arguments host, sh, -c, and exit 5. It concatenates them to a string and sends it to the remote host;
  3. On the remote host, ssh invokes a shell and passes it the string sh -c exit 5;
  4. The remote shell invokes sh and passes it the -c option, exit as the command string, and 5 as the command name.

Note that, if we add words after exit 5, they are just passed to sh as further arguments - no error related to them not being recognized by the shell:

$ ssh test sh -x -c 'exit 5' a b c; echo $?
+ exit
0

strace confirms that 5 is not part of the command string given to sh, here; it is an argument:

$ ssh test strace -e execve sh -c 'exit 5'; echo $?
execve("/usr/bin/sh", ["sh", "-c", "exit", "5"], 0x7ffc0d744c38 /* 14 vars */) = 0
+++ exited with 0 +++
0

In order to execute sh -c 'command' on a remote host as intended, we have to be sure to properly send it the quotes too:

$ ssh test "sh -x -c 'exit 5'"; echo $?
+ exit 5
5

To make it clear that quoting the whole remote command is not relevant to our current issue, we could just write:

$ ssh test sh -x -c "'exit 5'"; echo $?
+ exit 5
5

Escaping the inner quotes with backslashes, instead of quoting two times, would work as well.


A note about the command ssh host sh -c ':; exit 5' (from the comments to your question). What it does is:

$ ssh test sh -x -c ':; exit 5'; echo $?
+ :
5

That is, exit 5 is executed by the outer shell, not by sh. Again, to let sh exit with the desired code:

$ ssh test sh -x -c "':; exit 5'"; echo $?
+ :
+ exit 5
5
  • This answer adds detail to the one already given. Unfortunately, I can only officially "accept" one answer. The behaviour still feels non-intuitive to me. – Hermann Dec 19 '18 at 14:00
  • @Hermann No problem with the accept part, of course. Feel free to further point out what you feel less intuitive, answers can always be extended/improved. – fra-san Dec 19 '18 at 14:09
  • The answer is fine. It is just the combination of ssh itself not needing quotes, but double quoting being a necessity as quotes are considered by both, local and remote shell. – Hermann Dec 19 '18 at 14:12
  • A very informative use of strace. Any idea what's actually happening to the 5? None of the shells seems to be doing anything with the 5 without the double layer of quotes. – Tim Kennedy Dec 19 '18 at 14:51
  • 1
    @TimKennedy The 5 (and what follows, if you add other words) are the zeroth and following parameters to sh. They are just unused here, except for the zeroth one that is used as the command name - but giving the command a name is probably not what we want. You could actually use them, and you can see them, for example, changing exit in ssh test sh -x -c exit 5 into \'echo \"\$0\" \"\$@\" \', as in ssh test sh -x -c \'echo \"\$0\" \"\$@\" \' 5 a b c. The whole point is that, without proper quoting, the command string wanted by sh -c ends up being just the first word that follows. – fra-san Dec 19 '18 at 15:27
2

The other answers are good at answering the question in lieu of the examples given. My real-world application is more complicated and involves a series of scripts and sub-processes. Here is a boiled-down example script I want to execute:

#!/bin/bash
sub-process-that-fails
# store and echo returncode for debug purposes
rc=$?
echo $rc
exit $rc

Trying to make sure that the remotely executed shell was actually bash and not dash (as pointed out by @JeffSchaller), I tried calling the script like this:

~$ ssh -t -t host /bin/bash -x /srv/scripts/run.sh ; echo $?

Which led to this weird output:

+ sub-process-that-fails
+ rc=5
+ echo 5
5
+ exit 5
0

After hours of poking around, I noticed there was a trap 'kill 0' EXIT set in the .bashrc. This is done to kill all sub-processes in case bash is killed. bash's trace does not seem to display this trap's execution. I moved the trap into the wrapper script. Now I can see what actually is executed:

+ trap 'kill 0' EXIT
+ sub-process-that-fails
+ rc=5
5
+ echo 5
+ exit 5
+ kill 0
0

The remote shell exits with the last command's exit code. It's kill 0 and it exits with 0.

  • Also, if you just remove the trap, it works as expected, right? Similarly to ssh $somehost /bin/false setting the (client-side) exit status to 1? – ilkkachu Dec 19 '18 at 14:53
  • Yes. But the trap is there for a reason. Working around that is not the scope of this question, though. – Hermann Dec 19 '18 at 17:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.