Show whole output only when a process is complete

Question

Here is my script:

#! /bin/bash

ret=0
file_out1=$(mktemp)
file_out2=$(mktemp)

(echo first start; sleep 2s; echo first finish;) &> $file_out1 &
ret=$?
P1=$!
(echo second start; sleep 1s; echo second finish;) &> $file_out2 &
ret=$?
P2=$!

wait $P1 $P2 || ret=$?

cat $file_out1
cat $file_out2

exit $ret

Here, the output is:

first start
first finish
second start
second finish

But the output should be:

second start
second finish
first start
first finish

What I mean is, cat $file_out1 should run as soon as P1 is complete and cat $file_out2 should run as soon as P2 is complete.

Moreover, I want to make sure P1 ended successfully, and P2 ended successfully. At the end, how can I know if there was an error in P1 or P2 or both.

In this script, $ret is not working properly (it is always exit 0). After it works, It would be nice to know whether the $ret was from P1 or P2. Say something like P1 has return code x1, P2 has return code x2. And exit the program with a return code which is non-zero if there is a non-zero exit code.

Update1:

One answer in askubuntu, suggests that we can use --group argument of parallel command to ensures, that the output of each command waits till the command has finished (avoids mixed output).

The command it showed is parallel --group 'echo -n {};sleep {};echo {}' ::: 1 3 2 4.

This command works in my machine. However, I am not understanding how to use this command in my context.

I mean how can I get

second start
second finish
first start
first finish

When the input commands are echo first start; sleep 2s; echo first finish; and echo second start; sleep 1s; echo second finish; using gnu parallel.

Update 2:

The code I am currently at is:

#! /bin/bash

function first {
    echo first start
    sleep 2s
    echo first finish
}
function second {
    echo second start
    sleep 1s
    echo second finish
}

export -f first && export -f second

parallel -j2 ::: first second

This actually solves my problem.

Answer 1

I suppose you expect each of your task to show output as soon as they terminate, but avoid folding their output, that's why you bufferize it in a file - that's ok, you could also prefix output with sed, let it flow as it comes, then sort it again on demand : that would be in case of unpredictable quantity of data and risk of storage overflow.

Your mistakes :

• when you fork background processes with &, you lose access to their final status (what you expect in ret=$? is something else)
• when you wait for their completion, you cannot predict which one will terminate first, and whether you will wait on time to catch every process final status : too late after the process termination, wait will get 127 even if the process final status were 0. And that is unpredictable.

Your wish is to have the same feedback as running sequentially, but with the duration advantage of parallelism, and minimizing the sauce.

Let them execute in their own box, manage the race for output - flock on a file descriptor is clean - and force them to record their final status somewhere - again, a file descriptor is clean.

EDIT GNU parallel was all-in-one solution, as @AhmadIsmail edit say. Let this one anyway.

#!/bin/bash
# wrap background
mybackground(){
 tmp=$(mktemp) && trap "rm -f $tmp" RETURN
 exec {safe1}>&1 {safe2}>&2 1>$tmp 2>$tmp # swap descriptors
 (eval $@) # work
 echo $? >&$fdstatus # report status 
 exec >&$safe1 2>&$safe2 # restore descriptors
 flock 1 cat $tmp # report output
}
(
mybackground "echo first start; sleep 2s; echo first finish;exit 2" &
mybackground "echo second start; sleep 1s; echo second finish" &
) {fdstatus}> >(
  while read rc;do ((sum+=rc));done;echo sum=$sum;exit $sum ## collect & sum status
) | cat # wait {fdstatus} termination