Read / write to the same file descriptor with shell redirection

Question

I'm trying to understand file descriptors in the context of shell redirection.

Why can't I have cat read from FD 3, which is being written to by ls's STDOUT?

{ err=$(exec 2>&1 >&3; ls -ld /x /bin); exec 0<&3; out=$(cat); } 3>&1;

When try this, cat still wants to read from my keyboard.

If this can't be done, why not?

---

Differentiation: This question is about reading / writing to the same file descriptor, using the problem presented by Redirect STDERR and STDOUT to different variables without temporary files as an example.

Answer 1

In

{ err=$(exec 2>&1 >&3; ls -ld /x /bin); exec 0<&3; out=$(cat); } 3>&1

The { ... } 3>&1 clones the fd 1 to fd 3. That just means that fd 3 now points to the same resource (same open file description) as what fd 1 pointed to. If you ran that from a terminal, that will probably be a fd open in read+write mode to a terminal device.

After exec 0<&3, fds 0, 1, and 3 are all pointing to that same open file description (created when your terminal emulator opened the slave side of the pseudo-terminal pair it created before executing your shell in the case of the command run in the terminal case above).

Then in out=$(cat), for the process executing cat the $(...) changes fd 1 to the writing end of a pipe, while 0 is still the tty device. So cat will read from the terminal device, so things you're typing on the keyboard (and if it wasn't a terminal device, you would probably get an error as the fd was probably open in write-only mode).

For cat to read what ls writes on its stdout, you'd need ls stdout and cat stdin to be two ends on an IPC mechanism like pipe, socketpair or pseudo-terminal pair. For instance ls stdout to be the writing end of a pipe and cat stdin to be the reading end.

But you'd also need ls and cat to run concurrently, not one after the other, as that's an IPC (inter-process communication) mechanism.

Since pipes can hold some data (64 KiB by default on current versions of Linux), you would get away with short outputs if you managed to create that second pipe, but for larger outputs, you'd run into deadlocks, ls would hang when the pipe is full and would hang until something empties the pipe, but cat can only empty the pipe when ls returns.

Also, only yash has a raw interface to pipe() which you'd need to create that second pipe to read from ls stdout (the other pipe for stderr being created by the $(...) construct).

In yash, you'd do:

{ out=$(ls -d / /x 2>&3); exec 3>&-; err=$(exec cat <&4); } 3>>|4

Where 3>>|4 (a yash-specific feature) creates that second pipe with the writing end on fd 3 and the reading end on fd 4.

But again, if the stderr output is greater than the size of the pipe, that will hang. We're effectively using the pipe as a temporary file in memory, not a pipe.

To really use pipes, we'd need to start ls with stdout being the writing end of one pipe and stderr being the writing end of another pipe, and then the shell read the other ends of those pipes concurrently, as the data comes (not one after the other or again you'd run into dead-locks) to store into the two variables.

To be able to read from those two fds as the data comes, you'd need a shell with select()/poll() support. zsh is such a shell, but it doesn't have yash's pipeline redirection feature¹, so you'd need to use named pipes (so manage their creation, permissions, and cleanup) and use a complex loop with zselect/sysread...

¹ If on Linux though, you would be able to use the fact that /proc/self/fd/x on a pipe behaves like a named pipe though, so you could do:

#! /bin/zsh -
zmodload zsh/zselect
zmodload zsh/system

(){exec {wo}>$1 {ro}<$1} <(:) # like yash's wo>>|ro (but on Linux only)
(){exec {we}>$1 {re}<$1} <(:)

ls -d / /x >&$wo 2>&$we &
exec {wo}>&- {we}>&-
out= err=
o_done=0 e_done=0

while ((! (o_done && e_done))) && zselect -A ready $ro $re; do
  if ((${#ready[$ro]})); then
    sysread -i $ro && out+=$REPLY || o_done=1
  fi
  if ((${#ready[$re]})); then
    sysread -i $re && err+=$REPLY || e_done=1
  fi
done

Answer 2

It seems like a basic limitation of the variable assignment syntax, and the side-effects of the shell spawning subshells. You can either capture stderr or stdout but not both: the other stream needs to redirected into file (perhaps a FIFO).

# a function for testing
your_command() { sh -c 'echo "this is stdout"; echo "this is stderr" >&2'; }

errfile=$(mktemp)
out=$( your_command 2>|"$errfile" )
err=$(< "$errfile")
rm "$errfile"

echo "out: $out"
echo "err: $err"