25

Some shells, like bash, support Process Substitution which is a way to present process output as a file, like this:

$ diff <(sort file1) <(sort file2)

However, this construct isn't POSIX and, therefore, not portable. How can process substitution be achieved in a POSIX-friendly manner (i.e. one which works in /bin/sh) ?

note: the question isn't asking how to diff two sorted files - that is only a contrived example to demonstrate process substitution!

16

That feature was introduced by ksh (in ksh86) and was making use of the /dev/fd/n feature (added independently in some BSDs and AT&T systems earlier). In ksh and up to ksh93u, it wouldn't work unless your system had support for /dev/fd/n. zsh, bash and ksh93u+ and above can make use of temporary named pipes (named pipes added in SysIII I believe) where /dev/fd/n are not available.

On systems where /dev/fd/n is available (POSIX doesn't specify those), you can do process substitution yourself (diff <(cmd1) <(cmd2)) with:

{
  cmd1 4<&- | {
    # in here fd 3 points to the reading end of the pipe
    # from cmd1, while fd 0 has been restored from the original
    # stdin (saved on fd 4, now closed as no longer needed)

    cmd2 3<&- | diff /dev/fd/3 -

  } 3<&0 <&4 4<&- # restore the original stdin for cmd2

} 4<&0 # save a copy of stdin for cmd2

However that doesn't work with ksh93 on Linux as there, shell pipes are implemented with socketpairs instead of pipes and opening /dev/fd/3 where fd 3 points to a socket doesn't work on Linux.

Though POSIX doesn't specify /dev/fd/n. It does specify named pipes. Named pipes work like normal pipes except that you can access them from the file system. The issue here is that you have to create temporary ones and clean up afterwards which is hard to do reliably especially considering that POSIX has no standard mechanism (like a mktemp -d as found on some systems) to create temporary files or directories, and doing signal handling portably (to clean-up upon hang-up or kill) is also hard to do portably.

You could do something like:

tmpfifo() (
  n=0
  until
    fifo=$1.$$.$n
    mkfifo -m 600 -- "$fifo" 2> /dev/null
  do
    n=$((n + 1))
    # give up after 20 attempts as it could be a permanent condition
    # that prevents us from creating fifos. You'd need to raise that
    # limit if you intend to create (and use at the same time)
    # more than 20 fifos in your script
    [ "$n" -lt 20 ] || exit 1
  done
  printf '%s\n' "$fifo"
)

cleanup() { rm -f -- "$fifo"; }
fifo=$(tmpfifo /tmp/fifo) || exit

cmd2 > "$fifo" & cmd1 | diff - "$fifo"
rm -f -- "$fifo"

(not taking care of signal handling here).

  • The named pipe example is completely clear to me (I guess 10 is an arbitrary limit?) but I can't figure out your /dev/fd/n example. Why is descriptor 4 closed twice? (And so is descriptor 3.) I'm lost. – Wildcard Sep 13 '16 at 23:47
  • @Wildcard, it's closed for commands that don't need it. Here, only diff needs that fd 3. None need the fd 4, it's only used to propagate the original stdin to cmd2 (dup2(0,4) on the outer {...}, restored with dup2(4,0) for the inside {...}). – Stéphane Chazelas Sep 14 '16 at 7:28
  • You could use mktemp -d to help ensure you can get FIFOs, as nobody should be writing to your brand-new random temporary directory. – Daniel H Jan 27 '17 at 16:32
  • @DanielH, I already mention mktemp -d. But that's not a standard/POSIX command. – Stéphane Chazelas Jan 27 '17 at 17:18
  • Huh, I didn’t realize that. Oops. Quickly searching seems to show that most systems support it, so it might still be portable, but it isn’t POSIX. – Daniel H Feb 2 '17 at 19:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.