Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a bash shell script in which I pipe some data through about 5 or 6 different programs then the final results into a tab delimited file.

I then do the same again for a separate similar dataset and output to a second file.

Then both files are input into another program for comparative analysis. e.g. to simplify

Data1 | this | that |theother | grep |sed | awk |whatever > Data1Res.csv
Data2 | this | that |theother | grep |sed | awk |whatever > Data2Res.csv
AnalysisProg -i Data1res.csv Data2res.csv

My question is : how can I make step1 and step2 run at the same time (e.g. using &) but only launch step3 (AnalysisProg) when both are complete?

thx

ps AnalysisProg will not work on a stream or fifo.

share|improve this question
    
Check this: pebblesinthesand.wordpress.com/2008/05/22/… –  Bichoy Mar 28 at 15:26
    
BTW, is it ok for you to use Perl scripts? This can simplify the matter a lot for you and you can implement this post-processing very efficiently and make it run in parallel effortlessly. –  Bichoy Mar 28 at 15:27
    
Perl..not so much, no :( –  Stephen Henderson Mar 28 at 15:30
1  
Here i demonstrate how to split input across pipes with tee and process it with two concurrent grep processes: unix.stackexchange.com/questions/120333/… –  mikeserv Mar 29 at 11:33
    
And here i demonstrate how to use simple shell constructs to fully background a process in the way nohup might but still maintaining a means of communicating with the process: unix.stackexchange.com/questions/121253/… –  mikeserv Mar 29 at 11:34

4 Answers 4

up vote 12 down vote accepted

Use wait. For example:

Data1 ... > Data1Res.csv &
Data2 ... > Data2Res.csv &
wait
AnalysisProg

will:

  • run the Data1 and Data2 pipes as background jobs
  • wait for them both to finish
  • run AnalysisProg.

See, e.g., this question.

share|improve this answer
    
Thx, that looks good. I'll try this if the above doesn't work. –  Stephen Henderson Mar 28 at 15:33
    
Thx again, i was kind of aware of wait but having googled a bit was confused to how it worked with different PID etc.. I feel daft now I see it is just "wait" –  Stephen Henderson Mar 28 at 21:37

cxw's answer is no doubt the preferable solution, if you only have 2 files. If the 2 files are just examples and you in reality have 10000 files, then the '&' solution will not work, as that will overload your server. For that you need a tool like GNU Parallel:

ls Data* | parallel 'cat {} | this | that |theother | grep |sed | awk |whatever > {}res.csv
AnalysisProg -i *res.csv

To learn more about GNU Parallel:

share|improve this answer
    
Hi thx. At this time I do have two files, but I have 24 processors so I was feeling tempted to try and run many pairs at once - though as not a computing science person I'm unclear whether the disk reading bottleneck would make it worthwhile. maybe I'll suck it and see ;) –  Stephen Henderson Mar 28 at 21:34
    
@StephenHenderson depending on size the files can still be in cache. If speed's critical you can just use tmpfs (and the files are <<< then your RAM). –  Maciej Piechotka Mar 28 at 21:52
1  
@StephenHenderson The number of parallel jobs can be adjusted with -j, so try -j4 and if the server does not overload, try -j6 etc. But be ready to press CTRL-C: GNU Parallel is an excellent tool for overloading servers quickly. Also have a look at --load. –  Ole Tange Mar 28 at 22:50

Try use this.

rm -f Data1Res.csv
rm -f Data2Res.csv
Data1 | this | that |theother | grep |sed | awk |whatever > Data1Res.csv &
Data2 | this | that |theother | grep |sed | awk |whatever > Data2Res.csv &
while true
do
  ps aux | grep -v grep | grep -i -E 'Data1Res.csv|Data2Res.csv' &> /dev/null
  if [ $? -ne 0 ]
  then
    AnalysisProg -i Data1res.csv Data2res.csv
    exit 0
  fi
done
share|improve this answer
    
Well that's an heavy one. Isn't it like reinventing wait's wheel ? –  John WH Smith Mar 29 at 14:23

One way to do this could look something like :

AnalysisProg <<PREPROCESS /dev/stdin
$( 
{   process1=$( pipe | line | 1 >&2 & echo $! )
    process2=$( pipe | line | 2 >&2 & echo $! )
    while ps -p $process1 $process2 >/dev/null; do
        sleep 1
    done
} 2>&1
)
#END
PREPROCESS

In this way you background both pipelines but still wait for them to finish executing before combining their output into stdin which is evaluated in a here-document and handed to AnalysisProg. If you can use wait this is even better than the while ps loop, but, shell depending, wait can object if you instruct it to wait around on a process that is not a child of the current shell.

Also note that the above method will collate output - so both processes will be writing out at once. If you instead wanted them separate, or appended one to another possibly you could do:

AnalysisProg 3<<PREPROCESS /dev/fd/3 /dev/stderr
$(
process1=$(... >&2 ...) 2>/dev/fd/3
...
} 3>/dev/fd/3 2>/dev/stderr
)

I've demonstrated these concepts before. Probably the best demos are here and here.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.