Background processes in ksh script are disappearing after they are kicked off

Question

I'm having an issue with a KSH script that is supposed to kick off four processes in parallel in the background and then poll their status.

Put simply, running the four processes sequentially would take quite a bit of time, and I figure I can save a lot of time by running them in parallel. In theory I do not think that this will be hard to accomplish, and I've written the code below. Every process gets kicked off in the background and its pid gets stored in a variable.

Once all processes are kicked off, I want to check every pid using ps -p $PID to see if the process still exists. If it does not, it will exit the loop and move to the next one. This will be done in sequence.

Put aside that I still want to optimize the flow where possible, at this moment I am running into a very persistent problem of my processes disappearing. I start a process in the background, its pid get assigned to the variable (and I can verify that the corresponding variable is filled) but if lookup the process straight after using that same PID, nothing is returned. The process is just gone.

This happens with all 4 processes. Not running them in the background (so removing the ampersand) doesn't give me any problem whatsoever; the commands work fine. So the issue seems to be specific to running the desired commands as background processes.

If I run, for example, a sleep 180 in the background and store its PID in a similar way and query it the same way, I do get the desired result.

So I'm able to run the processes in the foreground and I am able to run other processes in the background. It's just that I am not able to run these specific processes in the background, for some reason unclear to me. Writing all output of the commands I want to run to a logfile returns empty log files. Using set -x doesn't show anything strange.

The script posted below calls upon a different script that does a number of checks and then calls upon a third script that contains a large number of functions. There's absolutely nothing I can find so far in the script that would indicate it causing trouble like this, and I run the exact same command lines I am kicking off below on the background in the shell directly, it starts normally.

So basically the situation is this:
- Running the desired commands on the foreground in this script works fine.
- Running another command (like sleep) on the background in this script works fine.
- Running the desired commands in the background directly in the shell works fine.
- Running the desired commands in the background in this script does not work and produces no recognisable errors.

At this point I'm completely lost about the issue. Could anyone give me at least a vague idea of why this won't work?

checkStatus(){
  LV_PID=${1}
  LV_COUNTER=0

  while (( ${LV_COUNTER} == 0 ))
  do
    ps -p ${LV_PID}
    if (( $? == 0 )); then
      ##continue loop
    elif (( $? == 1 )); then
      LV_COUNTER=1
    fi
  done
}

#################################################
#  Function: intendedFunction                   #
#################################################
intendedFunction(){
  nohup examplecommand.ksh -t exampletarget1 2> $HOME/log1.txt 1> /dev/null &
  export pid1=$!
  nohup examplecommand.ksh -t exampletarget2 2> $HOME/log2.txt 1> /dev/null &
  export pid2=$!
  nohup examplecommand.ksh -t exampletarget3 2> $HOME/log3.txt 1> /dev/null &
  export pid3=$!
  nohup examplecommand.ksh -t exampletarget4 2> $HOME/log4.txt 1> /dev/null &
  export pid4=$!

  echo "pid1 is $pid1"
  echo "pid2 is $pid2"
  echo "pid3 is $pid3"
  echo "pid4 is $pid4"

  checkStatus $pid1
  if (( `wc -l < log1.txt` == 0 ))
  then
    #success
  else
    #error
  fi
  checkStatus $pid2
  if (( `wc -l < log2.txt` == 0 ))
  then
    #success
  else
    #error
  fi
  checkStatus $pid3
  if (( `wc -l < log3.txt` == 0 ))
  then
    #success
  else
    #error
  fi
  checkStatus $pid4
  if (( `wc -l < log4.txt` == 0 ))
    #success
  else
    #error
  fi

Answer 1

I'd guess something basic, as in, a typo in the command name, or examplecommand.ksh not found in the path.

The pid variables will get filled by nohup, regardless of whether anything was really executed, and if nohup fails to run the command, it will exit immediately. Also, nohup won't bother you with any of those pesky error messages even if it utterly fails everything.

[ksh]$ nohup nonexistent.ksh &
[1] 32938
appending output to nohup.out
[ksh]$ echo $!
32938
[1] +  Done(127)               nohup nonexistent.ksh &

PS. You might want to take a look at xargs. As in

[ksh]$ seq 1 4 | xargs -n 1 -P 4 -I PART echo "part PART finished" 
part 2 finished
part 1 finished
part 3 finished
part 4 finished