coproc and named pipe behaviour under command substitution

Question

I have a requirement to make a function in a zsh shell script, that is called by command substitution, communicate state with subsequent calls to the same command substitution.

Something like C's static variables in functions (very crudely speaking).

To do this I tried 2 approaches - one using coprocessors, and one use named pipes. The named pipes approach, I can't get to work - which is frustrating because I think it will solve the only problem I have with coprocessors - that is, if I enter into a new zsh shell from the terminal, I don't seem to be able to see the coproc of the parent zsh session.

I've create simplified scripts to illustrate the issue below - if you're curious about what I'm trying to do - it's adding a new stateful component to the bullet-train zsh theme, that will be called by the command substituted build_prompt() function here: https://github.com/caiogondim/bullet-train.zsh/blob/d60f62c34b3d9253292eb8be81fb46fa65d8f048/bullet-train.zsh-theme#L692

Script 1 - Coprocessors

#!/usr/bin/env zsh

coproc cat
disown
print 'Hello World!' >&p

call_me_from_cmd_subst() {
    read get_contents <&p
    print "Retrieved: $get_contents"
    print 'Hello Response!' >&p
    print 'Response Sent!'
}

# Run this first
call_me_from_cmd_subst

# Then comment out the above call
# And run this instead
#print "$(call_me_from_cmd_subst)"

# Hello Response!
read finally <&p
echo $finally

Script 2 - Named Pipes

#!/usr/bin/env zsh

rm -rf /tmp/foo.bar
mkfifo /tmp/foo.bar
print 'Hello World!' > /tmp/foo.bar &

call_me_from_cmd_subst() {
    get_contents=$(cat /tmp/foo.bar)
    print "Retrieved: $get_contents"
    print 'Hello Response!' > /tmp/foo.bar &!
    print 'Response Sent!'
}

# Run this first
call_me_from_cmd_subst

# Then comment out the above call
# And run this instead
#print "$(call_me_from_cmd_subst)"

# Hello Response!
cat /tmp/foo.bar

In their initial forms they both produce exactly the same output:

$ ./named-pipe.zsh
Retrieved: Hello World!
Response Sent!
Hello Response!

$ ./coproc.zsh
Retrieved: Hello World!
Response Sent!
Hello Response!

Now if I switch the coproc script to call using the command substitution nothing changes:

# Run this first
#call_me_from_cmd_subst

# Then comment out the above call
# And run this instead
print "$(call_me_from_cmd_subst)"

That is reading and writing to the coprocess from the subprocess created by command substituion causes no issue. I was a little suprised by this - but it's good news!

But if I make the same change in the named piped examples the script blocks - with no output. To try to guage why I ran it with zsh -x, giving:

+named-pipe.zsh:3> rm -rf /tmp/foo.bar
+named-pipe.zsh:4> mkfifo /tmp/foo.bar
+named-pipe.zsh:15> call_me_from_cmd_subst
+call_me_from_cmd_subst:1> get_contents=+call_me_from_cmd_subst:1> cat /tmp/foo.bar
+named-pipe.zsh:5> print 'Hello World!'
+call_me_from_cmd_subst:1> get_contents='Hello World!'
+call_me_from_cmd_subst:2> print 'Retrieved: Hello World!'
+call_me_from_cmd_subst:4> print 'Response Sent!'

It looks to me like the subprocess created by the command substitution won't terminate whilst the following line hasn't terminated (I've played with using &, &!, and disown here with no change in result).

print 'Hello Response!' > /tmp/foo.bar &!

To demonstrate this I can manually fire-in a cat to read the response:

$ cat /tmp/foo.bar
Hello Response!

The script now waits at the final cat command as there is nothing in the pipe to read.

---

My questions are:

1. Is it possible to construct the named pipe to behave exactly like the coprocess in the presence of a command substitution?
2. Can you explain why a coprocess can demonstrably be read and written to from a subprocess, but if I manually create a subshell (by typing zsh) into the console, I can no longer access it (in fact I can create a new coproc that will operate independantly of its parent and exit, and continue using the parent's!).
3. If 1 is possible, I assume named pipes will have no such complicates as in 2 because the named pipe is not tied to a particular shell process?

To explain what I mean in 2 and 3:

$ coproc cat
[1] 24516
$ print -p test
$ read -ep
test
$ print -p test_parent
$ zsh
$ print -p test_child
print: -p: no coprocess
$ coproc cat
[1] 28424
$ disown
$ print -p test_child
$ read -ep
test_child
$ exit
$ read -ep
test_parent

I can't see the coprocess from inside the child zsh, yet I can see it from inside a command substitution subprocess?

Finally I'm using Ubuntu 18.04:

$ zsh --version
zsh 5.4.2 (x86_64-ubuntu-linux-gnu)

Answer 1

The reason your pipe-based script doesn't work isn't some peculiarity of zsh. It's due to the way shell command substitutions, shell redirections and pipes work. Here's the script without the superfluous parts.

mkfifo /tmp/foo.bar
echo 'Hello World!' > /tmp/foo.bar &

call_me_from_cmd_subst() {
    echo 'Hello Response!' > /tmp/foo.bar &
    echo 'Response Sent!'
}

echo "$(call_me_from_cmd_subst)"
cat /tmp/foo.bar

The command substitution $(call_me_from_cmd_subst) creates an anonymous pipe connecting the output of the subshell running the function to the original shell process. The original process reads from that pipe. The child process creates a grandchild process to run echo 'Hello Response!' > /tmp/foo.bar. Both processes start out with the same open files, including the anonymous pipe. The grandchild performs the redirection > /tmp/foo.bar. This blocks because nothing is reading from the named pipe /tmp/foo.bar.

Redirection is a two-step process (in fact three-step but the third doesn't matter here), because when you open a file, you don't get to choose its file descriptor. The > operator wants to redirect standard output, i.e. it wants to connect a specific file to file descriptor 1. This takes three system calls:

1. Call fd = open("/tmp/foo.bar", O_RDWR) to open the file. The file will be opened on some file descriptor fd that the process is not currently using. This is the step that blocks until something starts reading from the named pipe /tmp/foo.bar: opening a named pipe blocks if nobody is listening.
2. Call dup2(fd, 1) to open the file on the desired file descriptor in addition to the one the kernel chose. If there's anything open on the new descriptor (1), which there is (the anonymous pipe used for the command substitution), it is closed at this point.
3. Call close(fd), keeping the redirection target only on the desired file descriptor.

Meanwhile, the child prints Reponse Sent! and terminates. The original shell process is still reading from the pipe. Since the pipe is still open for writing in the grandchild, the original shell process keeps waiting.

To fix this deadlock, ensure that the grandchild does not keep the pipe open any longer than it has to. For example:

call_me_from_cmd_subst() {
    { exec >&-; /bin/echo 'Hello Response!' > /tmp/foo.bar; } &
    echo 'Response Sent!'
}

or

call_me_from_cmd_subst() {
    { echo 'Hello Response!' > /tmp/foo.bar; } >/dev/null &
    echo 'Response Sent!'
}

or any number of variations on this theme.

You don't have this problem with a coprocess because it doesn't involve a named pipe, so one of the halves of the deadlock isn't blocked: >/tmp/foo.bar blocks when it opens a named pipe, but >&p doesn't block since it's just redirecting an already-open file descriptor.

Answer 2

A very slight modification of your code works as expected:

#!/usr/bin/env zsh

rm -rf /tmp/foo.bar
mkfifo /tmp/foo.bar
print 'Hello World!' > /tmp/foo.bar &

call_me_from_cmd_subst() {
    get_contents=$(cat /tmp/foo.bar)
    print "Retrieved: $get_contents"
    (print 'Hello Response!' > /tmp/foo.bar &!) >/dev/null
    print 'Response Sent!'
}

# Run this first
#call_me_from_cmd_subst

# Then comment out the above call
# And run this instead
print "$(call_me_from_cmd_subst)"

# Hello Response!
cat /tmp/foo.bar

Running this yields the expected:

Retrieved: Hello World!
Response Sent!
Hello Response!

I'm not entirely sure why this is — it appears as if zsh believes the print to the fifo might result in some kind of output on stdout?