11

Suppose I want to execute multiple programs in parallel and combine their outputs to one pipe:

sh -c '
    (echo qqq; echo qqq2; echo qqq3)&
    (echo www; echo www2; echo www3)& 
    (echo eee; echo eee2; echo eee3)& 
  wait; wait; wait'

This shell approach works well for this simple case, but I expect it to fail if programs output more and longer lines in buffered way, like this (constructed):

qqq
qqwww
q2
qqq3www2

wwweee3

eee2
eee3

One of the solution I was hinted to use was tail -f:

tail -n +0 -q -f <(echo qqq; echo qqq2; echo qqq3) <(echo www; echo www2; echo www3) <(echo eee; echo eee2; echo eee3)

, but this is sub-optimal option: it outputs data sluggishly, it does not terminate; I see outputs not in "sleep" order, but in arguments order in this case:

tail -n +0 -q -f <(sleep 1; echo qqq; sleep 1; echo qqq2; echo qqq3) <(echo www; echo www2; sleep 10; echo www3) <(echo eee; sleep 4; echo eee2; echo eee3) | cat

I've implemented a special little program for this, but believe that there should be some standard good way to do it.

How to do it using standard tools (and without tail -f disadvantage)?

  • How do you want to mix the output? Apparently you do want to mix the output since you want “sleep order” rather than “arguments order”. Is your requirement to mix the output but not the lines, i.e. to have each line printed atomically? – Gilles Nov 27 '13 at 23:33
  • Linewise. All lines from all started programs should be delivered early, but without mixing inside each line. – Vi. Nov 28 '13 at 0:34
  • I think the standard way of doing this is called, well, syslog... – Shadur Nov 28 '13 at 6:12
  • Is using syslog not for logs, but for something custom considered OK? – Vi. Nov 28 '13 at 10:13
  • This isn't any more ideal than other suggestions posted thus far, but I thought it would be worth mentioning the -s option for tail. e.g. tail -f -s .1 file will reduce the loop delay to .1 seconds from the default 1 second. – cpugeniusmv Nov 29 '13 at 22:01
3

GNU Parallel.

From release notes dated August 2013:

--line-buffer will buffer output on line basis. --group keeps the output together for a whole job. --ungroup allows output to mixup with half a line coming from one job and half a line coming from another job. --line-buffer fits between these two; it prints a full line, but will allow for mixing lines of different jobs.

For example:

parallel --line-buffer <jobs

Where jobs contains:

./long.sh
./short.sh one
./short.sh two

short.sh:

#!/bin/bash

while true; do
        echo "short line $1"
        sleep .1
done

long.sh:

#!/bin/bash

count=0
while true; do
        echo -n "long line with multiple write()s "
        sleep .1
        count=$((count+1))
        if [ $count -gt 30 ]; then
                count=0
                echo
        fi
done

Output:

short line one
short line two
short line one
short line two
short line one
**-snip-**
short line one
short line one
short line two
short line two
short line one
short line one
short line one
long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s long line with multiple write()s 
short line two
short line two
short line two
short line one
1

A solution implementing locks :

function putlines () {
   read line || return $?
   while ! ln -s $$ lock >/dev/null 2>&1
   do
      sleep 0.05
   done
   echo "$line" 
}

function getlines () {
     while read lline
     do 
          echo "$lline"
          rm lock
     done
}

# your paralelized jobs  
(  
   job1 | putlines & 
   job2 | putlines & 
   job3 | putlines & 
   wait
) | getlines| final_processing

There should be a faster way to create a lock than using the filesystem.

0

I can't think of anything simple, that will help you, if your lines are so long, that one program will be sent to sleep before it was able, to finish writing a line to stdout.

However, if your lines are short enough to be written entirely before process switching, and your problem is, that generating one line takes very long, you can buffer the output using read.

E.g.:

((./script1 | while read line1; do echo $line1; done) & \
(./script2 | while read line2; do echo $line2; done)) | doSomethingWithOutput
  • Not beautiful. Unlikely that reliable. Unlikely that performance will be good. – Vi. Nov 29 '13 at 17:12
  • You're right. It's not beatiful but looks more like a dirty hack. However, I don't think thats enough to judge performance and reliability. In addition, you wanted to use 'standard tools'. So I wouldn't be suprised, if you have to accept some ugliness (in the end). But maybe someone has a more satisfactory solution. – xwst Nov 30 '13 at 0:18
  • Currently I'm satisfied with my program (linked in the question) except that it is not available in repositories thus can't be considered even a little "standard". The solution may be to trying to push it there... – Vi. Nov 30 '13 at 1:21
0

You could make a named pipe with mkfifo, dump all the output into the named pipe, and separately read from the named pipe for your collected data:

mkfifo /tmp/mypipe
job1 > /tmp/mypipe &
job2 > /tmp/mypipe &
job3 > /tmp/mypipe &

cat /tmp/mypipe > /path/to/final_output &

wait; wait; wait; wait
  • 1
    How will this protect from mangling when job1 and job2 output long (>4096 bytes) lines? This seems to be named pipe equivalent of the very code first example in quesion. – Vi. Dec 20 '13 at 20:40
  • Very fair point. I did not consider large-blob output despite it being called out explicitly in your question. I am now wondering if there is not perhaps some tool that does the reverse of tee, which sounds like exactly what you want. Possibly look at the internals of syslog or other logging tools, because they definitely aggregate output from several places into one log file. Locking may well be the right answer, as @emmanual suggested, also. – DopeGhoti Dec 20 '13 at 22:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.