I have a script which compares the output of a command with the output of the same command as it was ran previously, it works most of the time, but every now and then it doesn't work as expected.

I've been able to reproduce the issue in one test line. I know I could easily break this out into comparing two separate files and the problem would go away, but I'd like to understand what's actually happening here and if there's a way to achieve what I'm trying to achieve in the way I'm trying to achieve it.

Below is the output of my command ran several times, you can see the that it echoes "test" in one of the cases, but most of the time it works as expected.

root@dev:~# comm -13 /tmp/test <(echo '"test"' | cut -d'"' -f2 | sort -u | tee /tmp/test)
root@dev:~# comm -13 /tmp/test <(echo '"test"' | cut -d'"' -f2 | sort -u | tee /tmp/test)
root@dev:~# comm -13 /tmp/test <(echo '"test"' | cut -d'"' -f2 | sort -u | tee /tmp/test)
root@dev:~# comm -13 /tmp/test <(echo '"test"' | cut -d'"' -f2 | sort -u | tee /tmp/test)
root@dev:~# comm -13 /tmp/test <(echo '"test"' | cut -d'"' -f2 | sort -u | tee /tmp/test)
root@dev:~# comm -13 /tmp/test <(echo '"test"' | cut -d'"' -f2 | sort -u | tee /tmp/test)
root@dev:~# comm -13 /tmp/test <(echo '"test"' | cut -d'"' -f2 | sort -u | tee /tmp/test)
root@dev:~# comm -13 /tmp/test <(echo '"test"' | cut -d'"' -f2 | sort -u | tee /tmp/test)

I'm running Ubuntu 10.04, bash 4.1-2ubuntu3.5 and coreutils 7.4-2ubuntu3

  • 10.04 is no longer supported, but I'll let this one pass through. – eyoung100 Dec 18 '14 at 4:16
  • yeah, it's a race condition. – mikeserv Dec 18 '14 at 4:48
  • eyoung100, 10.04 is in LTS until April 2015. Unless you mean it's not supported here, in which case I understand :) – Nathan Dec 20 '14 at 8:37

Yes, that's a race condition.

The problem is that the shell starts all processes in the pipeline at the same time and tee truncates the output file on startup. If tee is faster then comm the file is empty for comm otherwise it is not.

The pipeline behaviour can be seen if you run this several times (mabe in a loop):

date '+first:  %T'|(cat>&2;sleep 5)|date '+second: %T'
  • Great, so how do I force tee to wait until all the previous commands in the pipeline are finished? – Nathan Dec 20 '14 at 8:36
  • I think the simplest way is to use a second file. I have no better idea at the moment. – Lucas Dec 21 '14 at 12:36

Your line confused me a bit as it was not clear what both occurrences of "test" are supposed to be. - Therefore I really understand that your shell is confused, too. ;)

If I understand correctly the first "/tmp/test" should correspond with the old output of the command and the second one corresponds with the new output.

You can be sure that stdin will contain the new output, but the content of the file is undefined: At the point comm reads that file, tee might not yet be started. In that case it will still contain the old "test" line from the last run. Or tee might already have finished. In that case it will contain the "test" line of the current run. Or tee just started, already cleared the file and is about to write the new content. If that happens you will compare the "test" from stdin with the empty file and you will see your output.

You have two different outputs you want to compare. I do not see a way to do this without having two different files. Having two different (properly named) files also makes the line much more readable and solves the confusion at the beginning. ;)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.