3

In a bash script I iterate through folders looking for some files, and if I find them I call a function with the directory that holds those files. See below

pairedread $1 &
pairedread $2 &
pairedread $3 & 
wait
echo "Done ..."

echo 
echo "======================"
echo "Testing again"
echo "======================"
echo

find . -type d -print | while read DIR; do
    echo "reading..."
    test -r "$DIR"/*_1.gz -a -r "$DIR"/*_2.gz || continue
    ( pairedread $DIR &  )
    done

wait
echo "Done..."

pairedread is the function that takes the folder and calls a python script on the files that are in the given directory. In the first case, ie when I explicitly give the folders that contain the files of interest, the script runs with instances of pairedread executing and finally terminating followed with the helpful message "Done..." after all subprocceses complete.

In the second case, the same three directories are picked and three instances of pairedread are created. However the script doesn't wait at all, it prints "Done..." immediately and returns while the subprocesses are running in the background.

Am I missing something? Why can't I wait for the subprocesses to finish before going on with the script?

  • Is there are reason for encapsulating the backgrounding with subshell in the second case? – Vikyboss Mar 29 '16 at 14:51
  • I'm going to guess that the wait in the outer doesn't know about the inner children since they're an extra layer removed because of the pipe. Try redoing the while to read from stdin the output of find and see if that helps? – Eric Renouf Mar 29 '16 at 14:51
2

Because you're launching those processes in a subshell (because of the pipe and because of the extra (...) options both) the wait doesn't know about any children to wait for. You can rewrite that loop so it doesn't require a subshell like:

while read DIR; do
    echo "reading..."
    test -r "$DIR"/*_1.gz -a -r "$DIR"/*_2.gz || continue
    pairedread $DIR &
done < <(find . -type d -print)

Wait only knows about children of the current process. When you use | it creates a subshell for the parts to connect the stdin/stdout of the sides together. Any process launched in a subshell is not a child of the "top" process, so wait doesn't know about them.

So in this case you were thwarted both by explicicly launching your pairedread in a subshell with () syntax, and also by having it happen within the block of a while loop in a pipeline.

Rerwriting that block to avoid the pipeline and removing the explicit subshells lets the outer wait know about the child processes and so do what you expected

1

As @Eric pointed out, you need to background outside the pipe and not subshell the command, if you want to wait on the process.

Double fork: When you do a minimum of two forks, the process will be orphaned and the init process becomes the parent of the process.

Your code: You did three forks(pipe, subshell and background), because of this, init process becomes parent of your newly started process and you cannot wait on it, as wait can only wait on its own child process.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.