How to make bidirectional pipe between two programs?

Question

Everyone knows how to make unidirectional pipe between two programs (bind stdout of first one and stdin of second one): first | second.

But how to make bidirectional pipe, i.e. cross-bind stdin and stdout of two programs? Is there easy way to do it in a shell?

Answer 1

Well, its fairly "easy" with named pipes (mkfifo). I put easy in quotes because unless the programs are designed for this, deadlock is likely.

mkfifo fifo0 fifo1
( prog1 > fifo0 < fifo1 ) &
( prog2 > fifo1 < fifo0 ) &
( exec 30<fifo0 31<fifo1 )      # write can't open until there is a reader
                                # and vice versa if we did it the other way

Now, there is normally buffering involved in writing stdout. So, for example, if both programs were:

#!/usr/bin/perl
use 5.010;
say 1;
print while (<>);

you'd expect a infinite loop. But instead, both would deadlock; you would need to add $| = 1 (or equivalent) to turn off output buffering. The deadlock is caused because both programs are waiting for something on stdin, but they're not seeing it because its sitting in the stdout buffer of the other program, and hasn't yet been written to the pipe.

Update: incorporating suggestions from Stéphane Charzelas and Joost:

mkfifo fifo0 fifo1
prog1 > fifo0 < fifo1 &
prog2 < fifo0 > fifo1

does the same, is shorter, and more portable.

Answer 2

If pipes on your system are bidirectional (as they are on Solaris 11 and some BSDs at least, but not Linux):

cmd1 <&1 | cmd2 >&0

Beware of deadlocks though.

Also note that some versions of ksh93 on some systems implement pipes (|) using a socket pair. socket pairs are bidirectional, but ksh93 explicitly shuts down the reverse direction, so the command above wouldn't work with those ksh93s even on systems where pipes (as created by the pipe(2) system call) are bidirectional.

Answer 3

I am not sure if this is what you are trying to do:

nc -l -p 8096 -c second &
nc -c first 127.0.0.1 8096 &

This begins by opening a listening socket on port 8096, and once a connection is established, spawns program second with its stdin as the stream output and stdout as the stream input.

Then, a second nc is launched which connects to the listening port and spawns program first with its stdout as the stream input and its stdin as the stream output.

This is not exactly done using a pipe, but it seems to do what you need.

As this uses the network, this can be done on 2 remote computers. This is almost the way a web server (second) and a web browser (first) work.

Answer 4

bash version 4 has coproc command that allows this done in pure bash without named pipes.

From bash man page about coproc command:

coproc [NAME] command [redirections]

This creates a coprocess named NAME. If NAME is not supplied, the default name is COPROC.

The standard output of command is connected via a pipe to a file descriptor in the executing shell, and that file descriptor is assigned to NAME[0]. The standard input of command is connected via a pipe to a file descriptor in the executing shell, and that file descriptor is assigned to NAME[1].

Start with a version easy to understand (but does not work). Start entire pipeline under coproc and use extra cat for plumbing.

coproc {
  cmd1 | cmd2
}
cat <&${COPROC[0]} >&${COPROC[1]}

Buffering is obviously something that needs to be taken care of. Here plain cat will likely break it due to buffering. So use stdbuf to disable buffering by cat:

# this is same as above
coproc {
  cmd1 | cmd2
}
# replace last line from sample above with:
stdbuf -i0 -o0 cat <&${COPROC[0]} >&${COPROC[1]}

Better still, do away with cat. Break up pipeline into two steps, run 1st section under coproc and 2nd section to connect with 1st.

coproc cmd1
cmd2 <&${COPROC[0]} >&${COPROC[1]}

This still assumes that cmd1 and cmd2 handle buffering correctly to make it work.

Some other shells also can do coproc as well.

That is all the answer above.

Below is just some exploration and tests.

Here example chains three commands only to make it a little more interesting.

# start pipeline
coproc {
    cmd1 | cmd2 | cmd3
}

# reconnect STDOUT from `cmd3` to STDIN of `cmd1`
stdbuf -i0 -o0 /bin/cat <&${COPROC[0]} >&${COPROC[1]}

Get rid of cat and stdbuf and stay with pure bash. Break it up into two parts, launch the first pipeline under coproc, then launch second part (either a single command or a pipeline), reconnecting it to the first:

coproc {
    cmd 1 | cmd2
}
cmd3 <&${COPROC[0]} >&${COPROC[1]}

Proof of concept

File prog - a worker program participating in looped IO. Just a dummy prog to consume, tag and re-print lines. It is using subshells to avoid buffering problems maybe overkill, it's not the point here.

#!/bin/bash

# Start this prog with only param "1" to produce some output

let c=0
sleep 2

[ "$1" == "1" ] && ( echo start )

while : ; do
  read line
  echo "$1:${c} ${line}" 1>&2
  sleep 2
  ( echo "$1:${c} ${line}" )
  let c++
  [ $c -eq 3 ] && exit
done

File start_io_loop_with_cat - a demo launcher. This is a version using bash, cat and stdbuf

#!/bin/bash

# start all 3 commands as one pipeline
coproc {
  ./prog 1 \
  | ./prog 2 \
  | ./prog 3
}

# start cat without buffering to connect IO loop
stdbuf -i0 -o0 /bin/cat <&${COPROC[0]} >&${COPROC[1]}

File start_io_loop_pure_bash - another demo launcher. This is a version using pure bash only.

#!/bin/bash

# start first 2 of 3 commands in the pipeline
coproc {
  ./prog 1 \
  | ./prog 2
}

# start 3rd command connecting IO to 1 and 2 to make the IO loop
./prog 3 <&${COPROC[0]} >&${COPROC[1]}

Output:

> ./start_io_loop_pure_bash
2:0 start
3:0 2:0 start
1:0 3:0 2:0 start
2:1 1:0 3:0 2:0 start
3:1 2:1 1:0 3:0 2:0 start
1:1 3:1 2:1 1:0 3:0 2:0 start
2:2 1:1 3:1 2:1 1:0 3:0 2:0 start
3:2 2:2 1:1 3:1 2:1 1:0 3:0 2:0 start
1:2 3:2 2:2 1:1 3:1 2:1 1:0 3:0 2:0 start

That does it.

Answer 5

You can use pipexec:

$ pipexec -- '[A' cmd1 ] '[B' cmd2 ] '{A:1>B:0}' '{B:1>A:0}'

Answer 6

A convenient building block for writing such bidirectional pipes is something that connects the stdout and stdin of the current process together. Let us call it ioloop. After calling this function, you only need to start a regular pipe:

ioloop &&     # stdout -> stdin 
cmd1 | cmd2   # stdin -> cmd1 -> cmd2 -> stdout (-> back to stdin)

If you do not want to modify the descriptors of the top-level shell, run this in a subshell:

( ioloop && cmd1 | cmd2 )

Here is a portable implementation of ioloop using a named pipe:

ioloop() {
    FIFO=$(mktemp -u /tmp/ioloop_$$_XXXXXX ) &&
    trap "rm -f $FIFO" EXIT &&
    mkfifo $FIFO &&
    ( : <$FIFO & ) &&    # avoid deadlock on opening pipe
    exec >$FIFO <$FIFO
}

The named pipe exists in the filesystem only briefly during ioloop setup. This function is not quite POSIX because mktemp is deprecated (and potentially vulnerable to a race attack).

A linux-specific implementation using /proc/ is possible that does not require a named pipe, but I think this one is more than good enough.

Answer 7

There is also

• dpipe, the "bi-directional pipe", included in vde2 package, and included in current distro package management systems.

  dpipe processA = processB

• socat, the connect everything-to-everthing tool.

  socat EXEC:Program1 EXEC:Program2

As @StéphaneChazelas correctly notes in the comments, above examples are the "base form", he has nice examples with options on his answer for a similar question.

Answer 8

There are many great answers here. So I just want to add something to easy play around with them. I assume stderr is not redirected anywhere. Create two scripts (let say a.sh and b.sh):

#!/bin/bash
echo "foo" # change to 'bar' in second file

for i in {1..10}; do
  read input
  echo ${input}
  echo ${i} ${0} got: ${input} >&2
done

Then when you connect them any good way you should see on the console:

1 ./a.sh got: bar
1 ./b.sh got: foo
2 ./a.sh got: foo
2 ./b.sh got: bar
3 ./a.sh got: bar
3 ./b.sh got: foo
4 ./a.sh got: foo
4 ./b.sh got: bar
...