5

I have a requirement to make a function in a zsh shell script, that is called by command substitution, communicate state with subsequent calls to the same command substitution.

Something like C's static variables in functions (very crudely speaking).

To do this I tried 2 approaches - one using coprocessors, and one use named pipes. The named pipes approach, I can't get to work - which is frustrating because I think it will solve the only problem I have with coprocessors - that is, if I enter into a new zsh shell from the terminal, I don't seem to be able to see the coproc of the parent zsh session.

I've create simplified scripts to illustrate the issue below - if you're curious about what I'm trying to do - it's adding a new stateful component to the bullet-train zsh theme, that will be called by the command substituted build_prompt() function here: https://github.com/caiogondim/bullet-train.zsh/blob/d60f62c34b3d9253292eb8be81fb46fa65d8f048/bullet-train.zsh-theme#L692

Script 1 - Coprocessors

#!/usr/bin/env zsh

coproc cat
disown
print 'Hello World!' >&p

call_me_from_cmd_subst() {
    read get_contents <&p
    print "Retrieved: $get_contents"
    print 'Hello Response!' >&p
    print 'Response Sent!'
}

# Run this first
call_me_from_cmd_subst

# Then comment out the above call
# And run this instead
#print "$(call_me_from_cmd_subst)"

# Hello Response!
read finally <&p
echo $finally

Script 2 - Named Pipes

#!/usr/bin/env zsh

rm -rf /tmp/foo.bar
mkfifo /tmp/foo.bar
print 'Hello World!' > /tmp/foo.bar &

call_me_from_cmd_subst() {
    get_contents=$(cat /tmp/foo.bar)
    print "Retrieved: $get_contents"
    print 'Hello Response!' > /tmp/foo.bar &!
    print 'Response Sent!'
}

# Run this first
call_me_from_cmd_subst

# Then comment out the above call
# And run this instead
#print "$(call_me_from_cmd_subst)"

# Hello Response!
cat /tmp/foo.bar

In their initial forms they both produce exactly the same output:

$ ./named-pipe.zsh
Retrieved: Hello World!
Response Sent!
Hello Response!

$ ./coproc.zsh
Retrieved: Hello World!
Response Sent!
Hello Response!

Now if I switch the coproc script to call using the command substitution nothing changes:

# Run this first
#call_me_from_cmd_subst

# Then comment out the above call
# And run this instead
print "$(call_me_from_cmd_subst)"

That is reading and writing to the coprocess from the subprocess created by command substituion causes no issue. I was a little suprised by this - but it's good news!

But if I make the same change in the named piped examples the script blocks - with no output. To try to guage why I ran it with zsh -x, giving:

+named-pipe.zsh:3> rm -rf /tmp/foo.bar
+named-pipe.zsh:4> mkfifo /tmp/foo.bar
+named-pipe.zsh:15> call_me_from_cmd_subst
+call_me_from_cmd_subst:1> get_contents=+call_me_from_cmd_subst:1> cat /tmp/foo.bar
+named-pipe.zsh:5> print 'Hello World!'
+call_me_from_cmd_subst:1> get_contents='Hello World!'
+call_me_from_cmd_subst:2> print 'Retrieved: Hello World!'
+call_me_from_cmd_subst:4> print 'Response Sent!'

It looks to me like the subprocess created by the command substitution won't terminate whilst the following line hasn't terminated (I've played with using &, &!, and disown here with no change in result).

print 'Hello Response!' > /tmp/foo.bar &!

To demonstrate this I can manually fire-in a cat to read the response:

$ cat /tmp/foo.bar
Hello Response!

The script now waits at the final cat command as there is nothing in the pipe to read.


My questions are:

  1. Is it possible to construct the named pipe to behave exactly like the coprocess in the presence of a command substitution?
  2. Can you explain why a coprocess can demonstrably be read and written to from a subprocess, but if I manually create a subshell (by typing zsh) into the console, I can no longer access it (in fact I can create a new coproc that will operate independantly of its parent and exit, and continue using the parent's!).
  3. If 1 is possible, I assume named pipes will have no such complicates as in 2 because the named pipe is not tied to a particular shell process?

To explain what I mean in 2 and 3:

$ coproc cat
[1] 24516
$ print -p test
$ read -ep
test
$ print -p test_parent
$ zsh
$ print -p test_child
print: -p: no coprocess
$ coproc cat
[1] 28424
$ disown
$ print -p test_child
$ read -ep
test_child
$ exit
$ read -ep
test_parent

I can't see the coprocess from inside the child zsh, yet I can see it from inside a command substitution subprocess?

Finally I'm using Ubuntu 18.04:

$ zsh --version
zsh 5.4.2 (x86_64-ubuntu-linux-gnu)
5
  • if a fifo is open for writing, it doesn't have an end. cat tries to read to the end, and blocks if there is no end. have you considered actually just reading one line at a time? – Fox Mar 8 at 17:13
  • I think cat will only block if there's nothing to read - and I'm pretty sure from the zsh logging that the it's not blocking on cat, it's the write blocking on termination of the command substitution's subprocess. When we remove the command substitution nothing blocks so the problem isn't the use of cat alone - it's something to do with the substitution. I think I could rephrase this as "how do I make the Hello Response print not block the command substitution exiting?" – Phil Mar 8 at 18:02
  • my statement was not one of guesswork. if a fifo is open for writing, then cat blocks on it when it reaches the end of what's there (unless there's enough unread to fill a buffer) – Fox Mar 8 at 19:50
  • Can you explain which cat you think is blocking in which script? I'm not disputing that this is fact - but I see no blocking without the command substitution, and with the exception of the last cat blocking on the named pipe after manual intervention (which is expected) the debug logs are not showing cat blocking (or I'm missing the point). – Phil Mar 8 at 20:00
  • apologies, my earlier responses were from a time when i was nowhere near a computer. having reached a real computer, i've come to an answer at least for (1) – Fox Mar 8 at 20:33
5

The reason your pipe-based script doesn't work isn't some peculiarity of zsh. It's due to the way shell command substitutions, shell redirections and pipes work. Here's the script without the superfluous parts.

mkfifo /tmp/foo.bar
echo 'Hello World!' > /tmp/foo.bar &

call_me_from_cmd_subst() {
    echo 'Hello Response!' > /tmp/foo.bar &
    echo 'Response Sent!'
}

echo "$(call_me_from_cmd_subst)"
cat /tmp/foo.bar

The command substitution $(call_me_from_cmd_subst) creates an anonymous pipe connecting the output of the subshell running the function to the original shell process. The original process reads from that pipe. The child process creates a grandchild process to run echo 'Hello Response!' > /tmp/foo.bar. Both processes start out with the same open files, including the anonymous pipe. The grandchild performs the redirection > /tmp/foo.bar. This blocks because nothing is reading from the named pipe /tmp/foo.bar.

Redirection is a two-step process (in fact three-step but the third doesn't matter here), because when you open a file, you don't get to choose its file descriptor. The > operator wants to redirect standard output, i.e. it wants to connect a specific file to file descriptor 1. This takes three system calls:

  1. Call fd = open("/tmp/foo.bar", O_RDWR) to open the file. The file will be opened on some file descriptor fd that the process is not currently using. This is the step that blocks until something starts reading from the named pipe /tmp/foo.bar: opening a named pipe blocks if nobody is listening.
  2. Call dup2(fd, 1) to open the file on the desired file descriptor in addition to the one the kernel chose. If there's anything open on the new descriptor (1), which there is (the anonymous pipe used for the command substitution), it is closed at this point.
  3. Call close(fd), keeping the redirection target only on the desired file descriptor.

Meanwhile, the child prints Reponse Sent! and terminates. The original shell process is still reading from the pipe. Since the pipe is still open for writing in the grandchild, the original shell process keeps waiting.

To fix this deadlock, ensure that the grandchild does not keep the pipe open any longer than it has to. For example:

call_me_from_cmd_subst() {
    { exec >&-; /bin/echo 'Hello Response!' > /tmp/foo.bar; } &
    echo 'Response Sent!'
}

or

call_me_from_cmd_subst() {
    { echo 'Hello Response!' > /tmp/foo.bar; } >/dev/null &
    echo 'Response Sent!'
}

or any number of variations on this theme.

You don't have this problem with a coprocess because it doesn't involve a named pipe, so one of the halves of the deadlock isn't blocked: >/tmp/foo.bar blocks when it opens a named pipe, but >&p doesn't block since it's just redirecting an already-open file descriptor.

8
  • i knew you'd eventually come along with an answer far better than mine! – Fox Mar 9 at 0:30
  • Is this right?: The grandchild holds the named pipe /tmp/foo.bar open for writing: echo 'Hello Response!' > /tmp/foo.bar & This will block until the /tmp/foo.bar pipe is read by cat. The cmd substiution is a different (?) named pipe created by the system, open for reading call_me_from_cmd_subst. The read on the cmd substitution blocks because the write is still open on /tmp/foo.bar inside call_me_from_cmd_subst and thus could contribute to stdout? So redirecting the waiting write to /dev/null tells the system this can't contribute to the read, and so breaks the deadlock? – Phil Mar 9 at 10:38
  • 1
    @Phil The grandchild doesn't hold the named pipe open: it's blocked trying to open it. The command substitution is on an anonymous pipe. The read on the command substitution blocks because the grandchild has this anonymous pipe still open for writing, it doesn't care about the named pipe. – Gilles 'SO- stop being evil' Mar 9 at 10:51
  • So any uncompleted command in call_me_from_cmd_subst could hold the anonymous pipe open for writing (the fact that the uncompleted command is the grandchild, blocked by a write to the named pipe is irrelevant from the point of view of the blocked read)? Explicitly redirecting the output of the whole grandchild process to /dev/null, means there cannot be anything to read from it, and thus unblocks the read on the anonymous pipe? Understanding what stdout could be redirected given the stdout of the echo is redirected to the named pipe is confusing! Is it just a requirement to be explicit? – Phil Mar 9 at 11:23
  • 1
    @Phil Correct. What matters is that nothing must have the command substitution pipe open for writing anymore. You can do that by closing the file descriptor explicitly (>&-) or by redirecting something onto it (>/dev/null). “given the stdout of the echo is redirected to the named pipe” — no, that's the point, the stdout of the echo is not yet redirected to the named pipe. If it was, the command substitution would end. But the redirection is blocked waiting for something to read from the named pipe, so the script just sits there waiting for a condition that never happens. – Gilles 'SO- stop being evil' Mar 9 at 12:02
3

A very slight modification of your code works as expected:

#!/usr/bin/env zsh

rm -rf /tmp/foo.bar
mkfifo /tmp/foo.bar
print 'Hello World!' > /tmp/foo.bar &

call_me_from_cmd_subst() {
    get_contents=$(cat /tmp/foo.bar)
    print "Retrieved: $get_contents"
    (print 'Hello Response!' > /tmp/foo.bar &!) >/dev/null
    print 'Response Sent!'
}

# Run this first
#call_me_from_cmd_subst

# Then comment out the above call
# And run this instead
print "$(call_me_from_cmd_subst)"

# Hello Response!
cat /tmp/foo.bar

Running this yields the expected:

Retrieved: Hello World!
Response Sent!
Hello Response!

I'm not entirely sure why this is — it appears as if zsh believes the print to the fifo might result in some kind of output on stdout?

4
  • Thanks v much - ignoring the reasons behind zsh's assumption that output may end up on stdout, can you clarify why output on stdout would be in an issue? Is it as simple as if the subprocess created by the command substitution was allowed to exit, this would close the stdout, and thus the print must hold it open until the print completes execution? – Phil Mar 8 at 20:56
  • 1
    A command substitution collects everything that was printed to stdout as a string. My analysis may be off, but it appears that zsh is waiting on that print to (possibly) put something on stdout — reading from the fifo allowed the print to exit and the command substitution to finish in your example, so explicitly telling zsh that it won't print to stdout seems to have solved things – Fox Mar 8 at 20:59
  • Yes of course - makes sense. It's a strange quirk given that print 'Hello Response!' >&p doesn't trigger the same paranoia. Redirecting to a coproc must follow a different code path to redirecting to a named pipe. – Phil Mar 8 at 21:03
  • @Phil No paranoia involved. Zsh isn't doing anything out of the ordinary. It's the combination of the way redirection works and the way named pipes work. Anonymous pipes work differently enough to not run into the problem. – Gilles 'SO- stop being evil' Mar 9 at 0:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.