What is the portable (POSIX) way to achieve process substitution?

Question

Some shells, like bash, support Process Substitution which is a way to present process output as a file, like this:

$ diff <(sort file1) <(sort file2)

However, this construct isn't POSIX and, therefore, not portable. How can process substitution be achieved in a POSIX-friendly manner (i.e. one which works in /bin/sh) ?

note: the question isn't asking how to diff two sorted files - that is only a contrived example to demonstrate process substitution!

Answer 1

That feature was introduced by ksh (first documented in ksh86) and was making use of the /dev/fd/n feature (added independently in some BSDs and AT&T systems earlier).  In ksh and up to ksh93u, it wouldn't work unless your system had support for /dev/fd/n. zsh, bash and ksh93u+ and above can make use of temporary named pipes (named pipes added in System III) where /dev/fd/n are not available.

On systems where /dev/fd/n is available (POSIX doesn't specify those), you can do process substitution (e.g., diff <(cmd1) <(cmd2)) yourself with:

{
  cmd1 4<&- | {
    # in here fd 3 points to the reading end of the pipe
    # from cmd1, while fd 0 has been restored from the original
    # stdin (saved on fd 4, now closed as no longer needed)

    cmd2 3<&- | diff /dev/fd/3 -

  } 3<&0 <&4 4<&- # restore the original stdin for cmd2

} 4<&0 # save a copy of stdin for cmd2

However that doesn't work with ksh93 on Linux as there, shell pipes are implemented with socketpairs instead of pipes and opening /dev/fd/3 where fd 3 points to a socket doesn't work on Linux.

Though POSIX doesn't specify /dev/fd/n, it does specify named pipes. Named pipes work like normal pipes except that you can access them from the file system. The issue here is that you have to create temporary ones and clean up afterwards, which is hard to do reliably, especially considering that POSIX has no standard mechanism (like a mktemp -d as found on some systems) to create temporary files or directories, and signal handling (to clean-up upon hang-up or kill) is also hard to do portably.

You could do something like:

tmpfifo() (
  n=0
  until
    fifo=$1.$$.$n
    mkfifo -m 600 -- "$fifo" 2> /dev/null
  do
    n=$((n + 1))
    # give up after 20 attempts as it could be a permanent condition
    # that prevents us from creating fifos. You'd need to raise that
    # limit if you intend to create (and use at the same time)
    # more than 20 fifos in your script
    [ "$n" -lt 20 ] || exit 1
  done
  printf '%s\n' "$fifo"
)

cleanup() { rm -f -- "$fifo"; }

fifo=$(tmpfifo /tmp/fifo) || exit

cmd2 > "$fifo" & cmd1 | diff - "$fifo"

cleanup

(not taking care of signal handling here).

Answer 2

You can rely on heredoc and /dev/fd/n files to achieve that. For example, in bash you can do this:

#!/usr/bin/env bash

paste  <(echo "$SHELL") <(echo "$TERM") <(echo "$LANG")

but in POSIX sh, you have to do this:

#!/bin/sh

paste /dev/fd/3 3<<-EOF /dev/fd/4 4<<-EOF /dev/fd/5 5<<-EOF 
$SHELL
EOF
$TERM
EOF
$LANG
EOF

The output is the same.

And to answer the question, this works:

#!/bin/sh

diff /dev/fd/3 3<<-EOF /dev/fd/4 4<<-EOF
$(sort file1)
EOF
$(sort file2)
EOF

It looks clunky, but it works.

Answer 3

If needed to avoid lost variable in the useful cmd | while read A B C, instead of:

VAR="before"
while read A B C 
do
  VAR="$A $VAR"
done < <(cmd)
echo "$VAR"

you can use:

VAR="before"
while read A B C 
do
  VAR="$A $VAR"
done << EndOfText
`cmd`
EndOfText
echo "$VAR"

So to answer the question:

sort file1 | diff /dev/stdin /dev/stdout 2<<EOT
`sort file2`
EOT