Communicate backwards in a pipe

Question

I have a simple pipeline:

node foo.js | node bar.js

bar.js will read from stdin to get data from foo.js.

But what I want to do is ensure that bar.js gets one of the last messages from foo.js before foo.js decides it's OK to exit. Essentially I want to create a simple request/response pattern.

foo writes to stdout --> bar reads from stdin --> how can bar send a message back to foo?

Is there a way to communicate backwards in a pipeline or should there never be a need to do that?

Answer 1

On systems where pipes are bidirectional (NetBSD, FreeBSD, SVR4-derived Unices (all those where pipes use STREAMS at least), but not Linux):

node foo.js <&1 | node bar.js >&0

Beside the named pipe already mentioned, you can also use a socketpair:

perl -MSocket -e '
  socketpair(A, B, AF_UNIX, SOCK_STREAM, PF_UNSPEC);
  if (fork) {
    open STDIN, "<&A";
    open STDOUT, ">&B";
    exec "node", "foo.js";
  } else {
    open STDIN, "<&B";
    open STDOUT, ">&A";
    exec "node", "bar.js";
  }'

Or two unnamed pipes, for instance using a coproc.

With zsh:

coproc node foo.js
node bar.js <&p >&p

ksh:

node foo.js |&
node bar.js <&p >&p

bash 4+:

coproc node foo.js
node bar.js <&"${COPROC[0]}" >&"${COPROC[1]}"

Or with yash's x>>|y pipe operator:

{ node foo.js <&3 3<&- | node bar.js 3<&-; } >>|3

Answer 2

No. A pipeline is a one-way communication channel. That's why it's called a "pipeline"; you couldn't send oil back up a pipeline if you tried, either.

However, if bar.js has to talk to foo.js too, you do have a few options:

• Create a unix domain socket instead of a pipeline, and start both foo.js and bar.js separately (i.e., don't pipe the output of foo.js into bar.js anymore). I don't know how you do that from node, but essentially a unix domain socket is a network socket that uses filenames rather than IP addresses and works inside the kernel. Sockets are meant for bidirectional communication, but require more setup than a simple pipe (e.g., a listening socket can talk to multiple instances of bar.js). You may find unix domain sockets in the filesystem, but it's not strictly necessary (and indeed Linux allows creating a unix domain socket without a trace on the filesystem).
• Use mkfifo to create a named pipe (or use some node API to create one, if that exists; again, I do not know node). Then, in foo.js, open that named pipe and read from it. Your bar.js script can open the same named pipe and write to it.

The latter will be easiest to transition to, since you still use file I/O (opening a named pipe requires opening a file on the filesystem), but would still be unidirectional (although you'd have two channels, one in each direction). The former is slightly cleaner and also allows you to more easily migrate one of the two scripts to a different host if that ever should become necessary.

At any rate, if your scripts are now communicating bidirectionally, then for clarity I would suggest you start them as separate processes, rather than having one process pipe into the other. IMHO, they're equal partners now, and your command line should show that. That's just a detail though, and certainly not technically required.

Answer 3

It's quite easy to communicate backwards, although probably not recommended. Of course you could get some really nice/unpleasant feedback loops in a complex system by communicating the wrong way.

On *nix you could do this with Node.js like so:

// foo.js
process.stdout.write(process.pid);

// bar.js
process.stdin.resume().once('data', function(pid){

   const writable = fs.createWriteStream(`/proc/${pid}/fd/0`);
   writable.write('whatevs');  // write to stdin of foo.js

});

the above is simplified, but you get the idea. Ideally you would use JSON to encode the stdio messages. Here's a good library to do that: https://github.com/ORESoftware/json-stdio

However, the problem is, on MacOS, you don't have ability to use /proc/<pid>... Last time I checked, the only good way to do that would be to use mkfifo to create a unique fifo for this situation, or to use TCP or Unix Domain Sockets.