17

I have a bash shell script in which I pipe some data through about 5 or 6 different programs then the final results into a tab delimited file.

I then do the same again for a separate similar dataset and output to a second file.

Then both files are input into another program for comparative analysis. e.g. to simplify

Data1 | this | that |theother | grep |sed | awk |whatever > Data1Res.csv
Data2 | this | that |theother | grep |sed | awk |whatever > Data2Res.csv
AnalysisProg -i Data1res.csv Data2res.csv

My question is : how can I make step1 and step2 run at the same time (e.g. using &) but only launch step3 (AnalysisProg) when both are complete?

thx

ps AnalysisProg will not work on a stream or fifo.

5

4 Answers 4

28

Use wait. For example:

Data1 ... > Data1Res.csv &
Data2 ... > Data2Res.csv &
wait
AnalysisProg

will:

  • run the Data1 and Data2 pipes as background jobs
  • wait for them both to finish
  • run AnalysisProg.

See, e.g., this question.

2
  • Thx, that looks good. I'll try this if the above doesn't work. Mar 28, 2014 at 15:33
  • Thx again, i was kind of aware of wait but having googled a bit was confused to how it worked with different PID etc.. I feel daft now I see it is just "wait" Mar 28, 2014 at 21:37
12

cxw's answer is no doubt the preferable solution, if you only have 2 files. If the 2 files are just examples and you in reality have 10000 files, then the '&' solution will not work, as that will overload your server. For that you need a tool like GNU Parallel:

ls Data* | parallel 'cat {} | this | that |theother | grep |sed | awk |whatever > {}res.csv
AnalysisProg -i *res.csv

To learn more about GNU Parallel:

3
  • Hi thx. At this time I do have two files, but I have 24 processors so I was feeling tempted to try and run many pairs at once - though as not a computing science person I'm unclear whether the disk reading bottleneck would make it worthwhile. maybe I'll suck it and see ;) Mar 28, 2014 at 21:34
  • @StephenHenderson depending on size the files can still be in cache. If speed's critical you can just use tmpfs (and the files are <<< then your RAM). Mar 28, 2014 at 21:52
  • 1
    @StephenHenderson The number of parallel jobs can be adjusted with -j, so try -j4 and if the server does not overload, try -j6 etc. But be ready to press CTRL-C: GNU Parallel is an excellent tool for overloading servers quickly. Also have a look at --load.
    – Ole Tange
    Mar 28, 2014 at 22:50
1

One way to do this could look something like :

AnalysisProg <<PREPROCESS /dev/stdin
$( 
{   process1=$( pipe | line | 1 >&2 & echo $! )
    process2=$( pipe | line | 2 >&2 & echo $! )
    while ps -p $process1 $process2 >/dev/null; do
        sleep 1
    done
} 2>&1
)
#END
PREPROCESS

In this way you background both pipelines but still wait for them to finish executing before combining their output into stdin which is evaluated in a here-document and handed to AnalysisProg. If you can use wait this is even better than the while ps loop, but, shell depending, wait can object if you instruct it to wait around on a process that is not a child of the current shell.

Also note that the above method will collate output - so both processes will be writing out at once. If you instead wanted them separate, or appended one to another possibly you could do:

AnalysisProg 3<<PREPROCESS /dev/fd/3 /dev/stderr
$(
process1=$(... >&2 ...) 2>/dev/fd/3
...
} 3>/dev/fd/3 2>/dev/stderr
)

I've demonstrated these concepts before. Probably the best demos are here and here.

0

Try use this.

rm -f Data1Res.csv
rm -f Data2Res.csv
Data1 | this | that |theother | grep |sed | awk |whatever > Data1Res.csv &
Data2 | this | that |theother | grep |sed | awk |whatever > Data2Res.csv &
while true
do
  ps aux | grep -v grep | grep -i -E 'Data1Res.csv|Data2Res.csv' &> /dev/null
  if [ $? -ne 0 ]
  then
    AnalysisProg -i Data1res.csv Data2res.csv
    exit 0
  fi
done
1
  • Well that's an heavy one. Isn't it like reinventing wait's wheel ? Mar 29, 2014 at 14:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.