4

I have a simple pipeline:

node foo.js | node bar.js

bar.js will read from stdin to get data from foo.js.

But what I want to do is ensure that bar.js gets one of the last messages from foo.js before foo.js decides it's OK to exit. Essentially I want to create a simple request/response pattern.

foo writes to stdout --> bar reads from stdin --> how can bar send a message back to foo?

Is there a way to communicate backwards in a pipeline or should there never be a need to do that?

  • 1
    I'm not sure what you're asking. The RHS of a pipeline will receive all data sent from the LHS (unless the LHS sends EOF, which should cause an error if more data are sent post-EOF as the pipe should then be closed). – DopeGhoti Nov 13 '17 at 17:25
  • The reason I ask is because I don't want foo to shutdown prematurely. (Probably won't happen but possible). Maybe one thing you could do - could you send a file descriptor or some address from foo.js to bar.js via stdio and then bar could write to foo that way? – Alexander Mills Nov 13 '17 at 17:48
  • Would there be a way to get a handle on stdin for foo.js and pass that information to bar.js? – Alexander Mills Nov 13 '17 at 17:50
  • Yeah, so there is no need to communicate backwards in my case - node bar.js won't exit until all data has been read from stdin, but the question still stands anyway, just for the heck of it. – Alexander Mills Nov 13 '17 at 18:40
3

No. A pipeline is a one-way communication channel. That's why it's called a "pipeline"; you couldn't send oil back up a pipeline if you tried, either.

However, if bar.js has to talk to foo.js too, you do have a few options:

  • Create a unix domain socket instead of a pipeline, and start both foo.js and bar.js separately (i.e., don't pipe the output of foo.js into bar.js anymore). I don't know how you do that from node, but essentially a unix domain socket is a network socket that uses filenames rather than IP addresses and works inside the kernel. Sockets are meant for bidirectional communication, but require more setup than a simple pipe (e.g., a listening socket can talk to multiple instances of bar.js). You may find unix domain sockets in the filesystem, but it's not strictly necessary (and indeed Linux allows creating a unix domain socket without a trace on the filesystem).
  • Use mkfifo to create a named pipe (or use some node API to create one, if that exists; again, I do not know node). Then, in foo.js, open that named pipe and read from it. Your bar.js script can open the same named pipe and write to it.

The latter will be easiest to transition to, since you still use file I/O (opening a named pipe requires opening a file on the filesystem), but would still be unidirectional (although you'd have two channels, one in each direction). The former is slightly cleaner and also allows you to more easily migrate one of the two scripts to a different host if that ever should become necessary.

At any rate, if your scripts are now communicating bidirectionally, then for clarity I would suggest you start them as separate processes, rather than having one process pipe into the other. IMHO, they're equal partners now, and your command line should show that. That's just a detail though, and certainly not technically required.

  • On many systems, Linux being a notable exception, pipes are bidirectional though. A socketpair() is a more portable way to have a bidirectional pipe (note that ksh93 uses a socketpair() for its | operator but shuts down the reverse direction). – Stéphane Chazelas Nov 13 '17 at 17:37
  • Could you send a file descriptor or some address from foo.js to bar.js via stdio and then bar could write to foo that way? – Alexander Mills Nov 13 '17 at 17:45
  • I added an answer to explain how you could do this with *nix, but not sure if it's as easy on MacOS...I have to support MacOS and on the latter, I think you need to create your own FIFO to do this. – Alexander Mills Nov 13 '17 at 18:36
  • @StéphaneChazelas a socketpair() is not a pipe, the two are not the same thing. A socketpair() is closer to a Unix Domain Socket instead. Also, do you have a source on your "pipes are bidirectional" claim? They're not meant to be. – Wouter Verhelst Nov 24 '17 at 11:49
  • @WouterVerhelst, no, socketpairs are not pipes, but can be used in places of pipes for the shell | (also called pipe) operator. That's what ksh93 does on Linux at least as said above). Pipes are bidirectional on at least FreeBSD and Solaris and I'd expect all SysV derived Unices and probably all BSDs, though I've only tested on FreeBSD and Solaris. That's clearly documented on FreeBSD and on Solaris – Stéphane Chazelas Nov 24 '17 at 12:29
3

On systems where pipes are bidirectional (NetBSD, FreeBSD, SVR4-derived Unices (all those where pipes use STREAMS at least), but not Linux):

node foo.js <&1 | node bar.js >&0

Beside the named pipe already mentioned, you can also use a socketpair:

perl -MSocket -e '
  socketpair(A, B, AF_UNIX, SOCK_STREAM, PF_UNSPEC);
  if (fork) {
    open STDIN, "<&A";
    open STDOUT, ">&B";
    exec "node", "foo.js";
  } else {
    open STDIN, "<&B";
    open STDOUT, ">&A";
    exec "node", "bar.js";
  }'

Or two unnamed pipes, for instance using a coproc.

With zsh:

coproc node foo.js
node bar.js <&p >&p

ksh:

node foo.js |&
node bar.js <&p >&p

bash 4+:

coproc node foo.js
node bar.js <&"${COPROC[0]}" >&"${COPROC[1]}"

Or with yash's x>>|y pipe operator:

{ node foo.js <&3 3<&- | node bar.js 3<&-; } >>|3
0

It's quite easy to communicate backwards, although probably not recommended. Of course you could get some really nice/unpleasant feedback loops in a complex system by communicating the wrong way.

On *nix you could do this with Node.js like so:

// foo.js
process.stdout.write(process.pid);

// bar.js
process.stdin.resume().once('data', function(pid){

   const writable = fs.createWriteStream(`/proc/${pid}/fd/0`);
   writable.write('whatevs');  // write to stdin of foo.js

});

the above is simplified, but you get the idea. Ideally you would use JSON to encode the stdio messages. Here's a good library to do that: https://github.com/ORESoftware/json-stdio

However, the problem is, on MacOS, you don't have ability to use /proc/<pid>... Last time I checked, the only good way to do that would be to use mkfifo to create a unique fifo for this situation.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.