I have a bash shell script in which I pipe some data through about 5 or 6 different programs then the final results into a tab delimited file.

I then do the same again for a separate similar dataset and output to a second file.

Then both files are input into another program for comparative analysis. e.g. to simplify

Data1 | this | that |theother | grep |sed | awk |whatever > Data1Res.csv
Data2 | this | that |theother | grep |sed | awk |whatever > Data2Res.csv
AnalysisProg -i Data1res.csv Data2res.csv

My question is : how can I make step1 and step2 run at the same time (e.g. using &) but only launch step3 (AnalysisProg) when both are complete?


ps AnalysisProg will not work on a stream or fifo.

  • Check this: pebblesinthesand.wordpress.com/2008/05/22/…
    – Bichoy
    Commented Mar 28, 2014 at 15:26
  • BTW, is it ok for you to use Perl scripts? This can simplify the matter a lot for you and you can implement this post-processing very efficiently and make it run in parallel effortlessly.
    – Bichoy
    Commented Mar 28, 2014 at 15:27
  • Perl..not so much, no :( Commented Mar 28, 2014 at 15:30
  • 1
    Here i demonstrate how to split input across pipes with tee and process it with two concurrent grep processes: unix.stackexchange.com/questions/120333/…
    – mikeserv
    Commented Mar 29, 2014 at 11:33
  • And here i demonstrate how to use simple shell constructs to fully background a process in the way nohup might but still maintaining a means of communicating with the process: unix.stackexchange.com/questions/121253/…
    – mikeserv
    Commented Mar 29, 2014 at 11:34

4 Answers 4


Use wait. For example:

Data1 ... > Data1Res.csv &
Data2 ... > Data2Res.csv &


  • run the Data1 and Data2 pipes as background jobs
  • wait for them both to finish
  • run AnalysisProg.

See, e.g., this question.

  • Thx, that looks good. I'll try this if the above doesn't work. Commented Mar 28, 2014 at 15:33
  • Thx again, i was kind of aware of wait but having googled a bit was confused to how it worked with different PID etc.. I feel daft now I see it is just "wait" Commented Mar 28, 2014 at 21:37

cxw's answer is no doubt the preferable solution, if you only have 2 files. If the 2 files are just examples and you in reality have 10000 files, then the '&' solution will not work, as that will overload your server. For that you need a tool like GNU Parallel:

ls Data* | parallel 'cat {} | this | that |theother | grep |sed | awk |whatever > {}res.csv
AnalysisProg -i *res.csv

To learn more about GNU Parallel:

  • Hi thx. At this time I do have two files, but I have 24 processors so I was feeling tempted to try and run many pairs at once - though as not a computing science person I'm unclear whether the disk reading bottleneck would make it worthwhile. maybe I'll suck it and see ;) Commented Mar 28, 2014 at 21:34
  • @StephenHenderson depending on size the files can still be in cache. If speed's critical you can just use tmpfs (and the files are <<< then your RAM). Commented Mar 28, 2014 at 21:52
  • 1
    @StephenHenderson The number of parallel jobs can be adjusted with -j, so try -j4 and if the server does not overload, try -j6 etc. But be ready to press CTRL-C: GNU Parallel is an excellent tool for overloading servers quickly. Also have a look at --load.
    – Ole Tange
    Commented Mar 28, 2014 at 22:50

One way to do this could look something like :

AnalysisProg <<PREPROCESS /dev/stdin
{   process1=$( pipe | line | 1 >&2 & echo $! )
    process2=$( pipe | line | 2 >&2 & echo $! )
    while ps -p $process1 $process2 >/dev/null; do
        sleep 1
} 2>&1

In this way you background both pipelines but still wait for them to finish executing before combining their output into stdin which is evaluated in a here-document and handed to AnalysisProg. If you can use wait this is even better than the while ps loop, but, shell depending, wait can object if you instruct it to wait around on a process that is not a child of the current shell.

Also note that the above method will collate output - so both processes will be writing out at once. If you instead wanted them separate, or appended one to another possibly you could do:

AnalysisProg 3<<PREPROCESS /dev/fd/3 /dev/stderr
process1=$(... >&2 ...) 2>/dev/fd/3
} 3>/dev/fd/3 2>/dev/stderr

I've demonstrated these concepts before. Probably the best demos are here and here.


Try use this.

rm -f Data1Res.csv
rm -f Data2Res.csv
Data1 | this | that |theother | grep |sed | awk |whatever > Data1Res.csv &
Data2 | this | that |theother | grep |sed | awk |whatever > Data2Res.csv &
while true
  ps aux | grep -v grep | grep -i -E 'Data1Res.csv|Data2Res.csv' &> /dev/null
  if [ $? -ne 0 ]
    AnalysisProg -i Data1res.csv Data2res.csv
    exit 0
  • Well that's an heavy one. Isn't it like reinventing wait's wheel ? Commented Mar 29, 2014 at 14:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .