bash script executed over ssh returns incorrect exit code 0

Question

I am trying to automate a process which involves running scripts on various machines via ssh. It is vital to capture both output and the return code (for the detection of errors).

Setting the exit code explicitly works as expected:

~$ ssh host exit 5 && echo OK || echo FAIL
FAIL

However, if there is a shell script signalling an unclean exit, ssh always returns 0 (script simulated by string execution):

~$ ssh host sh -c 'exit 5' && echo OK || echo FAIL
OK

Running the very same script on the host in an interactive shell works just fine:

~$ sh -c 'exit 5' && echo OK || echo FAIL
FAIL

I am confused as to why this happens. How can I tell ssh to propagate bash's return code? I may not change the remote scripts.

I am using public key authentication, the private key is unlocked – there is no need for user interaction. All systems are Ubuntu 18.04. Application versions are:

• OpenSSH_7.6p1 Ubuntu-4ubuntu0.1, OpenSSL 1.0.2n 7 Dec 2017
• GNU bash, Version 4.4.19(1)-release (x86_64-pc-linux-gnu)

Note: This question is different from these seemingly similar questions:

• bash shell - ssh remote script capture output and exit code?
• https://stackoverflow.com/questions/15390978/shell-script-ssh-command-exit-status
• https://stackoverflow.com/questions/36726995/exit-code-from-ssh-command
• https://superuser.com/questions/652729/command-executed-via-ssh-does-not-return-proper-return-code

Answer 1

As noted in the answer you already have, the remote sh is not executing exit 5. Just exit:

$ ssh test sh -x -c 'exit 5'; echo $?
+ exit
0

What is happening here is explained, for instance, in this answer:

ssh executes a remote shell and passes a string to it, not a list of arguments.

When we execute ssh host sh -c 'exit 5':

1. The local shell removes the single quotes (quote removal);
2. The ssh client gets the arguments host, sh, -c, and exit 5. It concatenates them to a string and sends it to the remote host;
3. On the remote host, ssh invokes a shell and passes it the string sh -c exit 5;
4. The remote shell invokes sh and passes it the -c option, exit as the command string, and 5 as the command name.

Note that, if we add words after exit 5, they are just passed to sh as further arguments - no error related to them not being recognized by the shell:

$ ssh test sh -x -c 'exit 5' a b c; echo $?
+ exit
0

strace confirms that 5 is not part of the command string given to sh, here; it is an argument:

$ ssh test strace -e execve sh -c 'exit 5'; echo $?
execve("/usr/bin/sh", ["sh", "-c", "exit", "5"], 0x7ffc0d744c38 /* 14 vars */) = 0
+++ exited with 0 +++
0

In order to execute sh -c 'command' on a remote host as intended, we have to be sure to properly send it the quotes too:

$ ssh test "sh -x -c 'exit 5'"; echo $?
+ exit 5
5

To make it clear that quoting the whole remote command is not relevant to our current issue, we could just write:

$ ssh test sh -x -c "'exit 5'"; echo $?
+ exit 5
5

Escaping the inner quotes with backslashes, instead of quoting two times, would work as well.

---

A note about the command ssh host sh -c ':; exit 5' (from the comments to your question). What it does is:

$ ssh test sh -x -c ':; exit 5'; echo $?
+ :
5

That is, exit 5 is executed by the outer shell, not by sh. Again, to let sh exit with the desired code:

$ ssh test sh -x -c "':; exit 5'"; echo $?
+ :
+ exit 5
5

Answer 2

I am able to duplicate this using the command you used, and I am able to resolve it by wrapping the remote command in quotes. Here are my test cases:

#!/bin/bash -x

echo 'Unquoted Test:'
ssh evil sh -x -c exit 5 && echo OK || echo FAIL

echo 'Quoted Test 1:'
ssh evil sh -x -c 'exit 5' && echo OK || echo FAIL

echo 'Quoted Test 2:'
ssh evil 'sh -x -c "exit 5"' && echo OK || echo FAIL

Here are the results:

bash-[540]$ bash -x test.sh
+ echo 'Unquoted Test:'
Unquoted Test:
+ ssh evil sh -x -c exit 5
+ exit
+ echo OK
OK
+ echo 'Quoted Test 1:'
Quoted Test 1:
+ ssh evil sh -x -c 'exit 5'
+ exit
+ echo OK
OK
+ echo 'Quoted Test 2:'
Quoted Test 2:
+ ssh evil 'sh -x -c "exit 5"'
+ exit 5
+ echo FAIL
FAIL

In the first test and second tests, it seems the 5 is not being passed to exit as we would expect it to be. It just seems to be disappearing. It's not going to exit, sh isn't complaining about 5: command not found, and ssh isn't complaining about it.

In the third test, exit 5 is quoted within the larger command to run on the remote host, same as in the second test. This ensures that the 5 is passed to exit, and both are executed as the -c option to sh. The difference between the second and third tests is that the whole set of commands and arguments is sent to the remote host quoted as a single command argument to ssh.

Answer 3

The other answers are good at answering the question in lieu of the examples given. My real-world application is more complicated and involves a series of scripts and sub-processes. Here is a boiled-down example script I want to execute:

#!/bin/bash
sub-process-that-fails
# store and echo returncode for debug purposes
rc=$?
echo $rc
exit $rc

Trying to make sure that the remotely executed shell was actually bash and not dash (as pointed out by @JeffSchaller), I tried calling the script like this:

~$ ssh -t -t host /bin/bash -x /srv/scripts/run.sh ; echo $?

Which led to this weird output:

+ sub-process-that-fails
+ rc=5
+ echo 5
5
+ exit 5
0

After hours of poking around, I noticed there was a trap 'kill 0' EXIT set in the .bashrc. This is done to kill all sub-processes in case bash is killed. bash's trace does not seem to display this trap's execution. I moved the trap into the wrapper script. Now I can see what actually is executed:

+ trap 'kill 0' EXIT
+ sub-process-that-fails
+ rc=5
5
+ echo 5
+ exit 5
+ kill 0
0

The remote shell exits with the last command's exit code. It's kill 0 and it exits with 0.