Everyone knows how to make unidirectional pipe between two programs (bind stdout of first one and stdin of second one): first | second.

But how to make bidirectional pipe, i.e. cross-bind stdin and stdout of two programs? Is there easy way to do it in a shell?

up vote 25 down vote accepted

If pipes on your system are bidirectional (as they are on Solaris 11 and some BSDs at least, but not Linux):

cmd1 <&1 | cmd2 >&0

Beware of deadlocks though.

Also note that some versions of ksh93 on some systems implement pipes (|) using a socket pair. socket pairs are bidirectional, but ksh93 explicitly shuts down the reverse direction, so the command above wouldn't work with those ksh93s even on systems where pipes (as created by the pipe(2) system call) are bidirectional.

Well, its fairly "easy" with named pipes (mkfifo). I put easy in quotes because unless the programs are designed for this, deadlock is likely.

mkfifo fifo0 fifo1
( prog1 > fifo0 < fifo1 ) &
( prog2 > fifo1 < fifo0 ) &
( exec 30<fifo0 31<fifo1 )      # write can't open until there is a reader
                                # and vice versa if we did it the other way

Now, there is normally buffering involved in writing stdout. So, for example, if both programs were:

#!/usr/bin/perl
use 5.010;
say 1;
print while (<>);

you'd expect a infinite loop. But instead, both would deadlock; you would need to add $| = 1 (or equivalent) to turn off output buffering. The deadlock is caused because both programs are waiting for something on stdin, but they're not seeing it because its sitting in the stdout buffer of the other program, and hasn't yet been written to the pipe.

Update: incorporating suggestions from Stéphane Charzelas and Joost:

mkfifo fifo0 fifo1
prog1 > fifo0 < fifo1 &
prog2 < fifo0 > fifo1

does the same, is shorter, and more portable.

  • 21
    One named pipe is sufficient: prog1 < fifo | prog2 > fifo. – Andrey Vihrov Nov 2 '12 at 16:47
  • 2
    @AndreyVihrov that's true, you can substitute an anonymous pipe for one of the named ones. But I like the symmetry :-P – derobert Nov 2 '12 at 17:26
  • 3
    @user14284: On Linux, you can probably do it with something like prog1 < fifo | tee /dev/stderr | prog2 | tee /dev/stderr > fifo. – Andrey Vihrov Nov 6 '12 at 10:28
  • 3
    If you make it prog2 < fifo0 > fifo1, you can avoid your little dance with exec 30< ... (which by the way only works with bash or yash for fds over 10 like that). – Stéphane Chazelas Apr 5 '14 at 20:08
  • 1
    @Joost Hmmm, appears you're correct that there is no need, at least in bash. I was probably worried that since the shell performs the redirects (including opening the pipes), it might block—but at least bash forks before opening the fifos. dash seems OK too (but behaves a little differently) – derobert Nov 9 '16 at 17:27

I am not sure if this is what you are trying to do:

nc -l -p 8096 -c second &
nc -c first 127.0.0.1 8096 &

This begins by opening a listening socket on port 8096, and once a connection is established, spawns program second with its stdin as the stream output and stdout as the stream input.

Then, a second nc is launched which connects to the listening port and spawns program first with its stdout as the stream input and its stdin as the stream output.

This is not exactly done using a pipe, but it seems to do what you need.

As this uses the network, this can be done on 2 remote computers. This is almost the way a web server (second) and a web browser (first) work.

You can use pipexec:

$ pipexec -- '[A' cmd1 ] '[B' cmd2 ] '{A:1>B:0}' '{B:1>A:0}'

bash version 4 has coproc command that allows this done in pure bash without named pipes:

coproc cmd1
eval "exec cmd2 <&${COPROC[0]} >&${COPROC[1]}"

Some other shells also can do coproc as well.

Below is more detailed answer but chains three commands, rather than two, which makes a only a little more interesting.

If you are happy to also use cat and stdbuf then construct can be made easier to understand.

Version using bash with cat and stdbuf, easy to understand:

# start pipeline
coproc {
    cmd1 | cmd2 | cmd3
}
# create command to reconnect STDOUT `cmd3` to STDIN of `cmd1`
endcmd="exec stdbuf -i0 -o0 /bin/cat <&${COPROC[0]} >&${COPROC[1]}"
# eval the command.
eval "${endcmd}"

Note, have to use eval because variable expansion in <&$var is illegal in my versio of bash 4.2.25.

Version using pure bash: Break up into two parts, launch first pipeline under coproc, then lunch second part, (either a single command or a pipeline) reconnecting it to the first:

coproc {
    cmd 1 | cmd2
}
endcmd="exec cmd3 <&${COPROC[0]} >&${COPROC[1]}"
eval "${endcmd}"

Proof of concept:

file ./prog, just a dummy prog to consume, tag and re-print lines. Using subshells to avoid buffering problems maybe overkill, it's not the point here.

#!/bin/bash
let c=0
sleep 2

[ "$1" == "1" ] && ( echo start )

while : ; do
  line=$( head -1 )
  echo "$1:${c} ${line}" 1>&2
  sleep 2
  ( echo "$1:${c} ${line}" )
  let c++
  [ $c -eq 3 ] && exit
done

file ./start_cat This is a version using bash, cat and stdbuf

#!/bin/bash

echo starting first cmd>&2

coproc {
  stdbuf -i0 -o0 ./prog 1 \
    | stdbuf -i0 -o0 ./prog 2 \
    | stdbuf -i0 -o0 ./prog 3
}

echo "Delaying remainer" 1>&2
sleep 5
cmd="exec stdbuf -i0 -o0 /bin/cat <&${COPROC[0]} >&${COPROC[1]}"

echo "Running: ${cmd}" >&2
eval "${cmd}"

or file ./start_part. This is a version using pure bash only. For demo purposes I am still using stdbuf because your real prog would have to deal with buffering internally anyway to avoid blocking due to buffering.

#!/bin/bash

echo starting first cmd>&2

coproc {
  stdbuf -i0 -o0 ./prog 1 \
    | stdbuf -i0 -o0 ./prog 2
}

echo "Delaying remainer" 1>&2
sleep 5
cmd="exec stdbuf -i0 -o0 ./prog 3 <&${COPROC[0]} >&${COPROC[1]}"

echo "Running: ${cmd}" >&2
eval "${cmd}"

Output:

> ~/iolooptest$ ./start_part
starting first cmd
Delaying remainer
2:0 start
Running: exec stdbuf -i0 -o0 ./prog 3 <&63 >&60
3:0 2:0 start
1:0 3:0 2:0 start
2:1 1:0 3:0 2:0 start
3:1 2:1 1:0 3:0 2:0 start
1:1 3:1 2:1 1:0 3:0 2:0 start
2:2 1:1 3:1 2:1 1:0 3:0 2:0 start
3:2 2:2 1:1 3:1 2:1 1:0 3:0 2:0 start
1:2 3:2 2:2 1:1 3:1 2:1 1:0 3:0 2:0 start

That does it.

A convenient building block for writing such bidirectional pipes is something that connects the stdout and stdin of the current process together. Let us call it ioloop. After calling this function, you only need to start a regular pipe:

ioloop &&     # stdout -> stdin 
cmd1 | cmd2   # stdin -> cmd1 -> cmd2 -> stdout (-> back to stdin)

If you do not want to modify the descriptors of the top-level shell, run this in a subshell:

( ioloop && cmd1 | cmd2 )

Here is a portable implementation of ioloop using a named pipe:

ioloop() {
    FIFO=$(mktemp -u /tmp/ioloop_$$_XXXXXX ) &&
    trap "rm -f $FIFO" EXIT &&
    mkfifo $FIFO &&
    ( : <$FIFO & ) &&    # avoid deadlock on opening pipe
    exec >$FIFO <$FIFO
}

The named pipe exists in the filesystem only briefly during ioloop setup. This function is not quite POSIX because mktemp is deprecated (and potentially vulnerable to a race attack).

A linux-specific implementation using /proc/ is possible that does not require a named pipe, but I think this one is more than good enough.

  • Interesting function, +1. Could probably use a sentence or 2 added explaining ( : <$FIFO & ) in more detail. Thank you for posting. – Alex Stragies Nov 9 '16 at 19:47
  • I did a little looking around and came up blank; where can I find out information about the deprecation of mktemp? I use it extensively, and if a newer tool has taken its place, I would like to start using it. – DopeGhoti Nov 9 '16 at 20:16
  • Alex: the open(2) syscall on a naned pipe is blocking. if you try to "exec <$PIPE >$PIPE" it will get stuck waiting for another process to open the other side. The command ": <$FIFO &" is executed in a subshell in the background and enables the bidirectional redirection to complete successfully. – user873942 Dec 4 '16 at 5:52
  • DopeGhoti: the mktemp(3) C library function is deprecated. The mktemp(1) utility is not. – user873942 Dec 4 '16 at 5:55

There is also

As @StéphaneChazelas correctly notes in the comments, above examples are the "base form", he has nice examples with options on his answer for a similar question.

  • Note that by default, socat uses sockets instead of pipes (you can change that with commtype=pipes). You may want to add the nofork option to avoid an extra socat process shoving data between the pipes/sockets. (thanks for the edit on my answer btw) – Stéphane Chazelas Jul 17 '16 at 15:27

There are many great answers here. So I just want to add something to easy play around with them. I assume stderr is not redirected anywhere. Create two scripts (let say a.sh and b.sh):

#!/bin/bash
echo "foo" # change to 'bar' in second file

for i in {1..10}; do
  read input
  echo ${input}
  echo ${i} ${0} got: ${input} >&2
done

Then when you connect them any good way you should see on the console:

1 ./a.sh got: bar
1 ./b.sh got: foo
2 ./a.sh got: foo
2 ./b.sh got: bar
3 ./a.sh got: bar
3 ./b.sh got: foo
4 ./a.sh got: foo
4 ./b.sh got: bar
...

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.