I have a problematic bash script and I really appreciate your help in identifying the problem. The bash script Script.sh is as follows:

./command1 -i inFile.txt -o outFile1.bin > log1.txt
./command2 -i inFile.txt -o outFile2.txt > log2.txt
./command3 -i outFile2.txt -o outFile3.txt > log3.txt

Here, -i defines the input file and -o defines the output file. There are some dependencies, for example the output file of command 2 is used as the input file for command 3.

I call the script using

nohup ./Script.sh > scriptLog.txt 2> scriptErr.txt &

Here is the strange thing that happens: the script runs to completion without any errors (scriptErr.txt is empty). The output files (outFile1.txt, outFile2.txt and outFile3.txt) are produced and are correct as much as I can say. The jobs command shows that the run is complete (Done). The command

ps aux | grep ./Script.sh

returns nothing (only the grep itself). However, the command

ps aux | grep ./command1

shows that command1 is still running (that is not the case about command2 and command3). And here is what confuses me:

This script is supposed to be sequential. How is it possible that commands 2 and 3 are completed while command 1 is not completed yet?

If command 1 is not completed, then how is that the output file is correctly produced?

Thank you very much!

  • 1
    Could command1 be running from any other source? Did you maybe run nohup command1 & in test a run and it is still stuck? Try the following: after each step, echo the PID of the previous command to a file and compare that to your ps aux | grep command1. Do they match? (PID of previous command: echo $!). Alternatively echo a confirmation string after each step :comman1 <...> && echo "command1 finished" – Fiximan Nov 11 '19 at 18:48
  • 2
    Are you positive that the grep ./command1 did not find the grep command itself? – Jeff Schaller Nov 11 '19 at 19:08
  • If you replace grep ./command1 with grep ./c[o]mmand1 is it still the same? – rush Nov 11 '19 at 19:27
  • You will have to give more information on command1. We know nothing about it, some commands go to background in certain conditions, so the behaviour could be completely normal. What does command1 do and how? (single command, another script, many commands on a pipe ...) – Eduardo Trápani Nov 11 '19 at 20:47
  • Thank you very much @Fiximan. That is one possibility and I will check it in the future runs. – sanaz Nov 13 '19 at 0:18

The problem here comes from nohup + &.

You use those to send the process to background. & also forks you current process. So instead of out Script.sh you get two (or multiplied by number of times you use &). New Script.sh process are children of the process you started. commandX is started from forked Script.sh.

Once main script process is finished, the parent Script.sh is gone, all the children are moved to init process. Those are different processes, just the name match and might introduce some confusion.

| improve this answer | |
  • Thank you very much @rush ! I am a little confused. You mean the parent process and the child process run simultaneously? Then, what happens if they are both writing to the same output file as is the case in my script? If that is the case, then what should I do if my script performs some critical writes and I want to run it in a remote server so that after I logout and login again it still continues running? – sanaz Nov 16 '19 at 17:39
  • @sanaz I meant there are multiple processes with the same name, but it doesn't mean they perform the same instructions and write to the same files (unless instructed so in code). In your case there will be multiple Script.sh processes, but those would control different children (command1, command2, command3, etc). – rush Nov 16 '19 at 18:24
  • Thank you very much @rush! – sanaz Nov 17 '19 at 0:46

It's easy to construct a script that stays running despite returning:

$ ./still_running 
I'm done
$ ps aux | grep '[s]till_running'
anthony  25666  0.0  0.0   7420  2092 pts/31   S    16:43   0:00 /bin/bash ./still_running

That script is fairly trivial:


    sleep 5m
    echo 'done now'
) &

echo "I'm done"

It starts a subshell, which it runs in the background (due to the the ampersand), and it stays running even when the parent process exits.

It's impossible to say for sure what's happening with your script without seeing the script (ideally a minimal test case), but this or something similar is a possibility.

| improve this answer | |
  • Thank you very much @derobert. Command1 runs an executable that converts a text file into a binary file (after processing it). It doesn't echo anything and it doesn't have any child processes. It is just a sequential C++ code. Also, the "Done" I mentioned was the status output by the command "jobs", it was not anything output by the command itself. Thank you again. – sanaz Nov 13 '19 at 0:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.