0

I may be misunderstanding what's going on here, but I feel it's got to do with pipe buffering. I have a script which uses several file descriptors (# 3 and above) for various logging levels. Depending on command line options, several of those may go to the same file, some to the console, some to /dev/null. I always have one of them (e.g. stdout) going to the file, and when I need to send several more to that file, I redirect them to stdout, rather than to the file. This is because I noticed that when I redirect several file descriptors to a file, they arrive out-of-order (which makes sense). That is to say I do

exec >/some/file 3>&1

rather than

exec >/some/file 3>/some/file

So far so good. However sometimes I need to take the error output of a command and send it to one of my custom descriptors, e.g. 3, which may be going to stdout (which in turn is going to a file). Then I get messages out of order. Messages from that command appear after messages from subsequent commands. Here's a small PoC. What am I doing wrong?

#!/bin/bash

check_if_ordered() {
  sort -n -k1 -k2 test_out.txt > test_out_sorted.txt
  if ! diff test_out.txt test_out_sorted.txt >/dev/null ; then
    echo "Oops, messages are NOT in order" >&2
  else
    echo "Good, messages are in order" >&2
  fi
  rm test_out.txt test_out_sorted.txt
}

log() {
  while read msg; do
    echo "$msg"
  done
}

foo() {
  for i in {1..150} ; do
    echo "$1 $i"
  done
}

#### This always works OK, but can't use it in my scenario
echo "Redirecting command output to file"
foo "1" > >(log) > test_out.txt
foo "2" > >(log) >> test_out.txt
check_if_ordered

#### This is similar to what I need to do and always fails
echo "Redirecting stdout to file"
exec >test_out.txt
foo "1" > >(log)
foo "2" > >(log)
check_if_ordered

I should add that I know about external tools which can disable buffering for a command, but I can't use those in this case (my script needs to be as portable as possible and run on various distros).

0
#### This always works OK, but can't use it in my scenario
foo "1" > >(log) > test_out.txt
foo "2" > >(log) >> test_out.txt

The process substitutions here seem redundant as the next redirections override the redirection of stdout straight to the file. That is, the first one should be equivalent to foo "1" > test_out.txt.

#### This is similar to what I need to do and always fails
exec >test_out.txt
foo "1" > >(log)
foo "2" > >(log)

Here, I think the issue is that the process substitution runs asynchronously, and since the while read; do echo loop is slow, the first log is still running and reading from the pipe buffers when the second starts. It's similar to doing something like echo foo > >(cat; sleep 1; echo hi) on the command line, the hi appears well after the next prompt is displayed. Also, wait doesn't seem to help here either.

But I'm not sure why you need two copies of log in the first place? Wouldn't just one do:

exec >test_out.txt
exec 9> >(log)
foo "1" >&9
foo "2" >&9

(On my system, using cat instead of while read; do echo also at least hides the problem, since cat is much faster and reads in larger blocks. But that doesn't mean it would completely fix it, and you probably want to do something other than just an identical copy in log.)

I should add that I know about external tools which can disable buffering for a command

If you mean stuff like stdbuf -o0, that only helps with buffering inside the process (in the C library), which I don't think comes into play here. The shell's echo should actually write the data out to the OS before returning.

| improve this answer | |
  • I see, I hadn't realised that process substitution is asynchronous :-| I really like your proposal to just run log once in the background using exec 9> >(log), that's way cleaner! Is there a way to redirect a FIFO file to log? Or should I just have log read from the FIFO instead of stdin and run it in the background? – Aayla Secura Oct 28 '19 at 22:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.