Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have (yet) another wait, &, && control flow question..

Say I have a script something like this where I want to do as much work at the same time as possible:

# may take some hours
something InputA > IrrelevantA &
something InputB > IrrelevantB &

# may take an hour
(
   somethingElse InputA > OutputA &
   somethingElse InputB > OutputB &
)&& combine OutputA OutputB > Result

...morestuff

Question 1: In the script, does combine wait for both somethingElse processes to finish while both something processes continue?

Question 2: If not - and I suspect it doesn't??? How do I get combine to wait only for both somethingElse processes whilst something processes above continue to work away in the background?

many thx (yet) again..

share|improve this question

3 Answers 3

In your example the combine command will just be run as soon as the subshell exits (and provided the last background process was started without an error). The subshell will exit immediately after the jobs are started since there is no wait command.

If you want to execute a command based on the return value of two or more simultaneous background processes, then I can't see any other way other than to use temporary files for the return values. This is because wait can only return the return value of one of the processes it waits for. Also since the background processes must be run in subshells to get their return values at all, they cannnot be stored in variables. You could do:

something InputA >IrrelevantA &
something InputB >IrrelevantB &

tmp1=$(mktemp)
tmp2=$(mktemp)

( somethingElse InputA >OutputA; echo $? >"$tmp1" ) &
proc1=$!

( somethingElse InputB >OutputB; echo $? >"$tmp2" ) &
proc2=$!

wait "$proc1" "$proc2"

read ret1 <"$tmp1"
read ret2 <"$tmp2"
[ "$ret1" = 0 && "ret2" = 0 ] && combine OutputA OutputB >Result

rm "$tmp1" "$tmp2"

If you don't really care about the return values, you can just start the jobs normally and use wait:

something InputA >IrrelevantA &
something InputB >IrrelevantB &

somethingElse InputA >OutputA &
proc1=$!

somethingElse InputB >OutputB &
proc2=$!

wait "$proc1" "$proc2"
combine OutputA OutputB >Result
share|improve this answer
    
Hi, I think the 2nd option would work for me... –  Stephen Henderson Apr 1 at 9:32

You can use the wait command:

(echo starting & sleep 10 & wait) && echo done

You can see the "starting" line happens right away, and the "done" waits for 10 seconds.

share|improve this answer
    
generally wait requires child processes of the same shell. Wait's pretty tricky there. –  mikeserv Apr 1 at 0:18
1  
@mikeserv, what are you talking about? That's the point: it waits for all of the children in that subshell. –  psusi Apr 1 at 1:06
    
by my initial tests this works. I'm going to try it on the big script now –  Stephen Henderson Apr 1 at 9:31
    
Exactly - children of the same shell - sub shells. It should work for any process that doesnt attempt to escape - or daemonize or whatever. Thats all i meant - as long as your processes respect process leaders wait's ok, but as soon as a process attempts to become its own process leader, wait will have issues. –  mikeserv Apr 1 at 17:51

I actually demonstrate exactly how this sort of thing might be done in another answer here. That answer was to a question about ensuring 2 logs were maintained by a background process, so I demonstrated it with 10.

Demo Script

cat <<-\DEMO >|${s=/tmp/script} 
printf 'tty is %s\nparent pid is %s\npid is pid=%s\n' \
     "$(tty)" "$PPID" "$$"
exec 1>&2 ; nums=$(seq 0 9)
rm ${files=$(printf "/tmp/file%s\n" $nums)}
for n in $nums ; do { for f in $files ; do
    echo "Line $n" >>"$f" ; done
sleep 1 ; } ; done
#END
DEMO

Run Demo

s=/tmp/script ;chmod +x $s ;info="$(($s &)2>&- &)"
echo "$info" ; pid="${info##*=}" ; echo
while ps -p $pid >/dev/null ; do sleep 3 ; done
for f in /tmp/file[0-9] ; do
    printf 'path : %s\tline count : %s\n' \
        $f $(<$f wc -l)
done

Output:

tty is not a tty
parent pid is 1
pid is 12123

path : /tmp/file0    line count : 10
path : /tmp/file1    line count : 10
path : /tmp/file2    line count : 10
path : /tmp/file3    line count : 10
path : /tmp/file4    line count : 10
path : /tmp/file5    line count : 10
path : /tmp/file6    line count : 10
path : /tmp/file7    line count : 10
path : /tmp/file8    line count : 10
path : /tmp/file9    line count : 10

The above demonstrates. It builds and runs a script named /tmp/script, chmod's it as executable, and runs it in the &background of a &backgrounded ( subshell ).

The script rms /tmp/file0-9 10 files and echoes a line every second into all 10 of them. I capture some $info from the disowned process and present it via $(command substitution). While ps still reports on the $pid I capture, I know it still runs so I sleep. When it completes, the lines in all 10 files are counted with wc.

After you invoke a process in this way you can freely close its original parent process and it will keep on trucking - it's effectively disowned. This also means you cannot use the conventional wait command, but waiting on ps's return should be more robust in any case.

Worth mentioning, I think, is that the process is actually initially called in $(command substitution) and printfs me the $info I want so I can effectively control it. But as soon as it drops its terminal output with exec 1>&2 (which is closed in the same subshell with 2>&-), the process escapes and I have to wait around for it on the other end. Kinda the best of both worlds, especially if you use it to handle input pipes, so long as you can wrap your mind around all of the redirections and process leaders.

Everything else is just for demonstration here. All you need to run this is the top script and:

info="$(($script_path &)2>&- &)"    

NOTE: This only prints to terminal exactly what I wished to demonstrate it. As noted by the $PPID, this process is disowned by the terminal and is a direct child of $PID 1.

If you wanted to run two of these concurrently and wait for them you could just hand ps both of their pids and wait.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.