Linux FIFO buffering depends on invocation order of readers and writers

Question

I'm trying to write a simple python script that reads from a fifo and then writes to another fifo.

I have created two FIFOs using the following command:

$ mkfifo input
$ mkfifo output

I invoke the script using the following command:

$ tail -f input | stdbuf -oL ../entropyCalc/entropy.py > output

, observe the FIFO output using:

$ tail -f output

and then invoke a writer using the following command:

$ echo "/path/to/a/valid/file" >> input

The problem is I expect the fifo to output the result as soon as it processes the input file, but I only observe that when I invoke (execute) the script once, exit and then re-execute the script. Everything works fine after this.

In summary:
I don't see any output in the reader if I execute script -> spin up reader -> write to input fifo
But, the command tail -f output outputs the result when I execute the script -> spin up the reader -> write to fifo -> kill script -> re-execute the script

I'm not sure what causes this behaviour since I'd expect the system to write to the files as soon as the result is written to stdout. I'd have expected the buffering if I would not used stdbuf -oL which limits buffering to a line.

The python script is a simple entropy calculator:

#! /usr/bin/env python2

import sys, os
import numpy as np
from scipy.stats import entropy

while 1:
    try:
        line = sys.stdin.readline()
    except KeyboardInterrupt:
        break

    if not line:
        break

    line = line.strip()

    if not line == '':
        fname1 = open(line)
        fsize = os.path.getsize(line)
        f1 = np.frombuffer(fname1.read(fsize), dtype=np.uint8)
        value,counts = np.unique(f1, return_counts=True)
        print line,str(entropy(counts))
        sys.stdout.flush()

I'm using bash 4.4 on Ubuntu 18.04.3

Answer 1

You used tail incorrectly.

What you observed is simply an indication that tail -f output works as advertised. And there are more pitfalls in your tail -f input usage.

Please note that these tail misuses will also break the operational integrity of your script. See below for details...

Short Explanation

Problems with your tail -f input usage:

• It waits for end-of-file at first.
• Once end-of-file is first reached (first filename-giver terminated), it would only pass last 10 filenames to the script; breaking operational integrity.
• It buffers until reaching each end-of-file, making interactive/sparse filename-entering unusable.

Problems with your tail -f output usage:

• It waits for end-of-file at first.
• Once end-of-file is first reached (your script got killed the first time), it would give only 10 last lines of the script output so far; breaking operational integrity.

(Buffering is not a problem at tail -f output since its output leg is a terminal)

See Conclusion below for remedies.

Long Explanation

The job description of tail is essentially: "output 10 lines that is just before end-of-file". The -f switch only adds "and everything that got subsequently appended to it".

• In order to accomplish its main goal, tail would have to keep reading the input, silently and endlessly (with only latest 10 lines kept in the buffer), until it got end-of-file. Then it would dump the last 10 lines from buffer to the output, and quit (or continue operating in -f phase).

Now, let me remind you that end-of-file on pipe never comes until the upstream program terminates¹...

The result is tail -f output will, by design, keep silently buffering through your script's result, without emitting anything... until the moment you first terminated your script: an end-of-file got sent to tail -f output, that's when you started seeing your output.

• But what you see is not the whole output so far, but rather just the last 10 lines of output at that point.

  If you asked your script to analyze 20 files (with less than 10 files at a time, see next point), you would get only the result of last 10 files analyzed just before you killed the script the first time. This breaks the expectation (i.e. integrity) of your script's operation, where result of every analysis done should have been reported on the output.

• This problem also affect tail -f input. Dumping a file list with 11+ filenames into your input pipe at first time, will result in only 10 bottommost names reaching your script, breaking the integrity of your script's operation again. On second and later times, this would not be a problem.

And that is not even the end of the story...

Once reached the first end-of-file, tail -f would continue operating in -f phase, which means output everything that got subsequently appended to the input file.

In this phase, it would not close the input handle yet; it will register an inotify listener on the input file, and wait. Each time that any program writes to/closes the input file, tail would read the input handle again (and output everything it just read), until it encounters end-of-file², and it will be back to waiting again. Rinse and repeat until you manually terminate tail -f.

• A catch here is "output everything it just read" also subjected to usual output buffering when the output is not terminal.

  It does not affect your use of tail -f output per se (as its output leg is a terminal). tail -f input however, will be affected:

  • If you entered multiple filenames interactively with cat > input , you would see that processing won't start until you terminate cat.
  • And this means connecting input pipe to a long-running program that only occasionally emit filenames, won't work either.

Conclusion

In my understanding, you used tail -f input to shield your script from receiving end-of-file, so your script could keep running like a daemon, while filename-giver programs could just come and go.

So my advice is to use tail -n +1 -f instead, to make sure that tail don't go on endless hunts for end-of-file, and eliminate the last-10-line problem. Then use stdbuf to control buffering where it would be an issue^3,4...

stdbuf -oL tail -n +1 -f input | ../entropyCalc/entropy.py > output 2>&1

Then on the monitoring side, use either:

tail -n +1 -f output

or:

while true; do cat output; done

Footnotes

• ¹ Or more precisely, when the upstream program closes the pipe. (In your case, it would only happen to tail -f output when your script terminates)
• ² Never happens to your tail -f output, unless you terminated your script the second time.
• ³ Your script is already line-buffered on its output stage, so no need for stdbuf there.
• ⁴ A standard error redirection 2>&1 will also ensure that script error also displays on the monitoring terminal.