How to run parallel processes and combine outputs when both finished

Question

I have a bash shell script in which I pipe some data through about 5 or 6 different programs then the final results into a tab delimited file.

I then do the same again for a separate similar dataset and output to a second file.

Then both files are input into another program for comparative analysis. e.g. to simplify

Data1 | this | that |theother | grep |sed | awk |whatever > Data1Res.csv
Data2 | this | that |theother | grep |sed | awk |whatever > Data2Res.csv
AnalysisProg -i Data1res.csv Data2res.csv

My question is : how can I make step1 and step2 run at the same time (e.g. using &) but only launch step3 (AnalysisProg) when both are complete?

thx

ps AnalysisProg will not work on a stream or fifo.

Answer 1

Use wait. For example:

Data1 ... > Data1Res.csv &
Data2 ... > Data2Res.csv &
wait
AnalysisProg

will:

• run the Data1 and Data2 pipes as background jobs
• wait for them both to finish
• run AnalysisProg.

See, e.g., this question.

Answer 2

cxw's answer is no doubt the preferable solution, if you only have 2 files. If the 2 files are just examples and you in reality have 10000 files, then the '&' solution will not work, as that will overload your server. For that you need a tool like GNU Parallel:

ls Data* | parallel 'cat {} | this | that |theother | grep |sed | awk |whatever > {}res.csv
AnalysisProg -i *res.csv

To learn more about GNU Parallel:

• Watch the intro video for a quick introduction: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
• Walk through the tutorial (man parallel_tutorial). You command line will love you for it.

Answer 3

One way to do this could look something like :

AnalysisProg <<PREPROCESS /dev/stdin
$( 
{   process1=$( pipe | line | 1 >&2 & echo $! )
    process2=$( pipe | line | 2 >&2 & echo $! )
    while ps -p $process1 $process2 >/dev/null; do
        sleep 1
    done
} 2>&1
)
#END
PREPROCESS

In this way you background both pipelines but still wait for them to finish executing before combining their output into stdin which is evaluated in a here-document and handed to AnalysisProg. If you can use wait this is even better than the while ps loop, but, shell depending, wait can object if you instruct it to wait around on a process that is not a child of the current shell.

Also note that the above method will collate output - so both processes will be writing out at once. If you instead wanted them separate, or appended one to another possibly you could do:

AnalysisProg 3<<PREPROCESS /dev/fd/3 /dev/stderr
$(
process1=$(... >&2 ...) 2>/dev/fd/3
...
} 3>/dev/fd/3 2>/dev/stderr
)

I've demonstrated these concepts before. Probably the best demos are here and here.

Answer 4

Try use this.

rm -f Data1Res.csv
rm -f Data2Res.csv
Data1 | this | that |theother | grep |sed | awk |whatever > Data1Res.csv &
Data2 | this | that |theother | grep |sed | awk |whatever > Data2Res.csv &
while true
do
  ps aux | grep -v grep | grep -i -E 'Data1Res.csv|Data2Res.csv' &> /dev/null
  if [ $? -ne 0 ]
  then
    AnalysisProg -i Data1res.csv Data2res.csv
    exit 0
  fi
done