Exit status of an asynchronous process

Question

Although the following command returns an exit status that depends on the existence of the remote file:

ssh $userAtServer "ls $targetDir/$targetFile" > $sshOutputFile
lsReturnValue=$?

which I can then test to do some stuff, it sometimes hangs (1 out of 10 or 20) and blocks further code execution.

So I need to run a ssh command like this one and to retrieve the exit value of the ls function:

(ssh $userAtServer "ls $targetDir/$targetFile" > $sshOutputFile ; lsReturnValue=$?) &
timeOutProcess $!

However, lsReturnValue always return an empty string.

timeOutProcess is a function that kills my command if it lasts for too much time:

timeOutProcess() {
    processId=$1

    #from http://www.bashcookbook.com/bashinfo/source/bash-4.0/examples/scripts/timeout3
    timeout=45
    interval=2
    delay=5
    (
        ((t = timeout))

        while ((t > 0)); do
            sleep $interval
            kill -0 $processId || return 0
            ((t -= interval))
        done
        # Be nice, post SIGTERM first.
        # The 'exit 0' below will be executed if any preceeding command fails.
        kill -s SIGTERM $processId && kill -0 $processId || exit 0
        sleep $delay
        kill -s SIGKILL $processId
    ) 2> /dev/null  
}

I wonder how could I get the $? value from the ssh command ?

Answer 1

You have two ways of obtaining the return status of the ssh process using standard shell features: either run it synchronously, or call the wait builtin. When you run

(ssh $userAtServer "ls $targetDir/$targetFile" > $sshOutputFile ; lsReturnValue=$?) &

this only sets lsReturnValue in the subshell, and you can't get any information other than the return status back to the parent shell. So you'd have to run exit $lsReturnValue from the subshell and obtain the subshell's return status from the parent shell; and then the whole background process could be simplified to ssh $userAtServer "ls $targetDir/$targetFile" > $sshOutputFile & anyway.

This shell snippet you found to allow a command to time out isn't a very good one because it doesn't retain the return status. On Linux, use the timeout utility.

Without the timeout utility, it gets a little tricky. You need to run two subprocesses: the one you're interested in, and one that runs sleep for the timeout part. If either process returns, it must kill the other one and allow the shell to proceed. Unfortunately, the wait builtin waits until all subprocesses have died when you call it with no argument. One way to synchronize the two processes is to make them both write to a pipe and kill them once some data appears on the pipe.

lsReturnValue=$(
  {
    { ssh $userAtServer "ls $targetDir/$targetFile" > $sshOutputFile
      echo $?; } &
    { sleep 5; echo timeout; } &
  } | head -n 1)
if [ "$lsReturnValue" = "timeout" ]; then …

If you're sending commands to multiple servers, seriously consider using an ssh multi-server framework such as mussh, clusterssh, etc.

Answer 2

Your command is getting hanged at times because of the OpenSSH feature which prevents SSH session from closing until all IO opened by process running in the SSH session are closed properly.

One way to prevent this is to redirect your STDIN and STDOUT to some thing other that the terminal.

ssh $userAtServer "ls $targetDir/$targetFile > /dev/null < /dev/null 2>&1"

# lsReturnValue=$?

If you need the STDOUT/STDERR of the command you can redirect them to a file and cat it later

ssh $userAtServer "ls $targetDir/$targetFile > /tmp/ls_output < /dev/null 2>&1;cat /tmp/ls_output"