Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Is there a tool that will take input from multiple files or pipes and write it to stdout, without blocking on reads, such that individual input lines come out intact? I basically want to multiplex a bunch of inputs onto one output without clobbering lines.

$ combine file1 <(prog2) ... > nice-output.txt
  1. I don't care about the order of the output
  2. It should not block as long as some input has data
  3. It should be efficient (i.e., I may downvote your Perl one-liner ;)
share|improve this question
add comment

2 Answers

You should be able to do this with multitail pretty easily.

share|improve this answer
    
Can you suggest what arguments I would use with multitail? It does not seem to have a non-interactive mode, hangs trying to write to stdout, and crashes reading from a pipe. –  Jay Hacker Jun 23 '11 at 20:20
    
Start with -L to run a command and merge the output with the current stream and -a to write the output to a file. I'll look more tomorrow. If you give a more detailed example I'll try to work it in. –  Caleb Jun 23 '11 at 20:23
add comment

If the processes write the lines in a single write call, which requires the processes to use line buffering (usually turned off if their standard output is not a terminal), you can just point them all to a pipe.

{ { sleep .1; echo one; sleep .1; echo two; } &
  { echo hello; sleep .15; echo world; };
  wait; } | cat

If the processes only perform line buffering when writing to a terminal, the easy way is to use script. It's a bit clumsy: it can only write to a file.

script -q -c '
    { { sleep .1; echo one; sleep .1; echo two; } &
      { echo hello; sleep .15; echo world; };
      wait; }'
tail -n +2 typescript

If the programs write long lines or just don't use line buffering, this approach isn't going to work. You'll need a collector program that reads and buffers lines from each input separately and performs the synchronization on line endings. There's no standard utility with this functionality. I second Caleb's suggestion of multitail.

Here's a Python script that reads lines produced by several commands and spits them out on its standard output, without breaking up a line. I haven't tested it much, so caveat user. I haven't benchmarked it at all.

#!/usr/bin/env python
import Queue, itertools, os, subprocess, sys, threading
# Queue of (producer_id, line). line==None indicates the end of a producer.
lq = Queue.Queue()

# Line producer
def run_task(i, cmd):
    p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
    line = p.stdout.readline()
    while line <> "":
        lq.put((i, line))
        line = p.stdout.readline()
    lq.put((i, None))

# Start a producer for each command passed as an argument
for i in range(1,len(sys.argv)):
    threading.Thread(target=run_task, args=(i, sys.argv[i])).start()
sources = len(sys.argv) - 1
# Consumer: print lines as they come in, until no producer is left.
while sources > 0:
    (k, line) = lq.get()
    if line == None: sources -= 1
    else: sys.stdout.write(str(k) + ":" + line)

Sample usage:

./collect.py 'sleep 1; ls /; sleep 1; ls /' \
             '/bin/echo -n foo; sleep 1; /bin/echo -n bar; sleep 1; /bin/echo qux'
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.