13

I have a bash shell script in which I pipe some data through about 5 or 6 different programs then the final results into a tab delimited file.

I then do the same again for a separate similar dataset and output to a second file.

Then both files are input into another program for comparative analysis. e.g. to simplify

Data1 | this | that |theother | grep |sed | awk |whatever > Data1Res.csv
Data2 | this | that |theother | grep |sed | awk |whatever > Data2Res.csv
AnalysisProg -i Data1res.csv Data2res.csv

My question is : how can I make step1 and step2 run at the same time (e.g. using &) but only launch step3 (AnalysisProg) when both are complete?

thx

ps AnalysisProg will not work on a stream or fifo.

25

Use wait. For example:

Data1 ... > Data1Res.csv &
Data2 ... > Data2Res.csv &
wait
AnalysisProg

will:

  • run the Data1 and Data2 pipes as background jobs
  • wait for them both to finish
  • run AnalysisProg.

See, e.g., this question.

  • Thx, that looks good. I'll try this if the above doesn't work. – Stephen Henderson Mar 28 '14 at 15:33
  • Thx again, i was kind of aware of wait but having googled a bit was confused to how it worked with different PID etc.. I feel daft now I see it is just "wait" – Stephen Henderson Mar 28 '14 at 21:37
12

cxw's answer is no doubt the preferable solution, if you only have 2 files. If the 2 files are just examples and you in reality have 10000 files, then the '&' solution will not work, as that will overload your server. For that you need a tool like GNU Parallel:

ls Data* | parallel 'cat {} | this | that |theother | grep |sed | awk |whatever > {}res.csv
AnalysisProg -i *res.csv

To learn more about GNU Parallel:

  • Hi thx. At this time I do have two files, but I have 24 processors so I was feeling tempted to try and run many pairs at once - though as not a computing science person I'm unclear whether the disk reading bottleneck would make it worthwhile. maybe I'll suck it and see ;) – Stephen Henderson Mar 28 '14 at 21:34
  • @StephenHenderson depending on size the files can still be in cache. If speed's critical you can just use tmpfs (and the files are <<< then your RAM). – Maciej Piechotka Mar 28 '14 at 21:52
  • 1
    @StephenHenderson The number of parallel jobs can be adjusted with -j, so try -j4 and if the server does not overload, try -j6 etc. But be ready to press CTRL-C: GNU Parallel is an excellent tool for overloading servers quickly. Also have a look at --load. – Ole Tange Mar 28 '14 at 22:50
1

One way to do this could look something like :

AnalysisProg <<PREPROCESS /dev/stdin
$( 
{   process1=$( pipe | line | 1 >&2 & echo $! )
    process2=$( pipe | line | 2 >&2 & echo $! )
    while ps -p $process1 $process2 >/dev/null; do
        sleep 1
    done
} 2>&1
)
#END
PREPROCESS

In this way you background both pipelines but still wait for them to finish executing before combining their output into stdin which is evaluated in a here-document and handed to AnalysisProg. If you can use wait this is even better than the while ps loop, but, shell depending, wait can object if you instruct it to wait around on a process that is not a child of the current shell.

Also note that the above method will collate output - so both processes will be writing out at once. If you instead wanted them separate, or appended one to another possibly you could do:

AnalysisProg 3<<PREPROCESS /dev/fd/3 /dev/stderr
$(
process1=$(... >&2 ...) 2>/dev/fd/3
...
} 3>/dev/fd/3 2>/dev/stderr
)

I've demonstrated these concepts before. Probably the best demos are here and here.

0

Try use this.

rm -f Data1Res.csv
rm -f Data2Res.csv
Data1 | this | that |theother | grep |sed | awk |whatever > Data1Res.csv &
Data2 | this | that |theother | grep |sed | awk |whatever > Data2Res.csv &
while true
do
  ps aux | grep -v grep | grep -i -E 'Data1Res.csv|Data2Res.csv' &> /dev/null
  if [ $? -ne 0 ]
  then
    AnalysisProg -i Data1res.csv Data2res.csv
    exit 0
  fi
done
  • Well that's an heavy one. Isn't it like reinventing wait's wheel ? – John WH Smith Mar 29 '14 at 14:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.