I have (yet) another wait, &, && control flow question..

Say I have a script something like this where I want to do as much work at the same time as possible:

# may take some hours
something InputA > IrrelevantA &
something InputB > IrrelevantB &

# may take an hour
   somethingElse InputA > OutputA &
   somethingElse InputB > OutputB &
)&& combine OutputA OutputB > Result


Question 1: In the script, does combine wait for both somethingElse processes to finish while both something processes continue?

Question 2: If not -- and I suspect it doesn't -- how do I get combine to wait only for both somethingElse processes whilst something processes above continue to work away in the background?

4 Answers 4


In your example the combine command will just be run as soon as the subshell exits (and provided the last background process was started without an error). The subshell will exit immediately after the jobs are started since there is no wait command.

If you want to execute a command based on the return value of two or more simultaneous background processes, then I can't see any other way other than to use temporary files for the return values. This is because wait can only return the return value of one of the processes it waits for. Also since the background processes must be run in subshells to get their return values at all, they cannnot be stored in variables. You could do:

something InputA >IrrelevantA &
something InputB >IrrelevantB &


( somethingElse InputA >OutputA; echo $? >"$tmp1" ) &

( somethingElse InputB >OutputB; echo $? >"$tmp2" ) &

wait "$proc1" "$proc2"

read ret1 <"$tmp1"
read ret2 <"$tmp2"
[ "$ret1" = 0 && "ret2" = 0 ] && combine OutputA OutputB >Result

rm "$tmp1" "$tmp2"

If you don't really care about the return values, you can just start the jobs normally and use wait:

something InputA >IrrelevantA &
something InputB >IrrelevantB &

somethingElse InputA >OutputA &

somethingElse InputB >OutputB &

wait "$proc1" "$proc2"
combine OutputA OutputB >Result
  • Hi, I think the 2nd option would work for me... Apr 1, 2014 at 9:32

You can use the wait command:

(echo starting & sleep 10 & wait) && echo done

You can see the "starting" line happens right away, and the "done" waits for 10 seconds.

  • generally wait requires child processes of the same shell. Wait's pretty tricky there.
    – mikeserv
    Apr 1, 2014 at 0:18
  • 1
    @mikeserv, what are you talking about? That's the point: it waits for all of the children in that subshell.
    – psusi
    Apr 1, 2014 at 1:06
  • by my initial tests this works. I'm going to try it on the big script now Apr 1, 2014 at 9:31
  • Exactly - children of the same shell - sub shells. It should work for any process that doesnt attempt to escape - or daemonize or whatever. Thats all i meant - as long as your processes respect process leaders wait's ok, but as soon as a process attempts to become its own process leader, wait will have issues.
    – mikeserv
    Apr 1, 2014 at 17:51

Would process substitution be more efficient, especially if you don't need to save the files OutputA and OutputB, and care only about Result? Would this be particularly time-saving because if you have a slow I/O in writing to disk, saving the files OutputA and OutputB might be the rate-limiting step?

combine  <(somethingElse InputA)  <(somethingElse InputB)  >  Result

Process substitution allows you to put the command inside <(..here..) instead of saving the output into a file, and then reading from it as input in the "combine" step.

If memory is a limitation, and the size of outputA and outputB more than what the memory can hold, will it defeat the whole purpose?

Will combine wait till both processes are completed before it starts running?

  • This is not “Jeopardy”; please do not phrase your response in the form of a question. Seriously, you have come up with a new idea, and I think it’s pretty good. To respond to a couple of your points: combine will start running as soon as the two somethingElse commands have started, but that’s OK, because the <(…) things are pipes; so combine will simply be forced to wait for data if it outruns the somethingElse processes. And, because they are pipes, size isn’t an issue. … (Cont’d) Mar 15, 2015 at 5:28
  • (Cont’d) … The only substantive problem I have with your answer is that it doesn’t allow for testing the exit status of the somethingElse processes – and it’s not entirely clear whether that’s important to the asker. But, also, an answer shouldn’t be asking questions like that. Mar 15, 2015 at 5:29

I actually demonstrate exactly how this sort of thing might be done in another answer here. That answer was to a question about ensuring 2 logs were maintained by a background process, so I demonstrated it with 10.

Demo Script

cat <<-\DEMO >|${s=/tmp/script} 
printf 'tty is %s\nparent pid is %s\npid is pid=%s\n' \
     "$(tty)" "$PPID" "$$"
exec 1>&2 ; nums=$(seq 0 9)
rm ${files=$(printf "/tmp/file%s\n" $nums)}
for n in $nums ; do { for f in $files ; do
    echo "Line $n" >>"$f" ; done
sleep 1 ; } ; done

Run Demo

s=/tmp/script ;chmod +x $s ;info="$(($s &)2>&- &)"
echo "$info" ; pid="${info##*=}" ; echo
while ps -p $pid >/dev/null ; do sleep 3 ; done
for f in /tmp/file[0-9] ; do
    printf 'path : %s\tline count : %s\n' \
        $f $(<$f wc -l)


tty is not a tty
parent pid is 1
pid is 12123

path : /tmp/file0    line count : 10
path : /tmp/file1    line count : 10
path : /tmp/file2    line count : 10
path : /tmp/file3    line count : 10
path : /tmp/file4    line count : 10
path : /tmp/file5    line count : 10
path : /tmp/file6    line count : 10
path : /tmp/file7    line count : 10
path : /tmp/file8    line count : 10
path : /tmp/file9    line count : 10

The above demonstrates. It builds and runs a script named /tmp/script, chmod's it as executable, and runs it in the &background of a &backgrounded ( subshell ).

The script rms /tmp/file0-9 10 files and echoes a line every second into all 10 of them. I capture some $info from the disowned process and present it via $(command substitution). While ps still reports on the $pid I capture, I know it still runs so I sleep. When it completes, the lines in all 10 files are counted with wc.

After you invoke a process in this way you can freely close its original parent process and it will keep on trucking - it's effectively disowned. This also means you cannot use the conventional wait command, but waiting on ps's return should be more robust in any case.

Worth mentioning, I think, is that the process is actually initially called in $(command substitution) and printfs me the $info I want so I can effectively control it. But as soon as it drops its terminal output with exec 1>&2 (which is closed in the same subshell with 2>&-), the process escapes and I have to wait around for it on the other end. Kinda the best of both worlds, especially if you use it to handle input pipes, so long as you can wrap your mind around all of the redirections and process leaders.

Everything else is just for demonstration here. All you need to run this is the top script and:

info="$(($script_path &)2>&- &)"    

NOTE: This only prints to terminal exactly what I wished to demonstrate it. As noted by the $PPID, this process is disowned by the terminal and is a direct child of $PID 1.

If you wanted to run two of these concurrently and wait for them you could just hand ps both of their pids and wait.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .