3

How can I close all readers of a FIFO once all its input is read? Seems I can only get one of them to close, which keeps my program from finishing.

Here's a sample program that works (to test put it into a file):

set -euo pipefail

rm -f todo.pipe
mkfifo todo.pipe

rm -f output.pipe
mkfifo output.pipe

cat todo.pipe | \
    while read line && echo hej $line; do :; done \
            > output.pipe &

echo "adam\n bertil\n carl" > todo.pipe &

cat < output.pipe

Output is, as expected:

❯ ./test.zsh
hej adam
hej bertil
hej carl
❯

However, if I add another thread to process the things off todo.pipe, things hang forever:

set -euo pipefail

rm -f todo.pipe
mkfifo todo.pipe

rm -f output.pipe
mkfifo output.pipe

cat todo.pipe | \
    while read line && echo hej $line; do :; done \
            > output.pipe &
# The below 3 lines is all that's changed
cat todo.pipe | \
    while read line && echo hej $line; do :; done \
            > output.pipe &

echo "adam\n bertil\n carl" > todo.pipe &

cat < output.pipe

Now, it prints the same as before, but it never returns. Why? And how can I get around this?

My suspicion is that it's something with the second "worker thread" now getting an EOF or something like that, but feels like I'm missing something fundamental here.

1
  • Both threads should get EOF when there are no writers of the pipe left. I suspect the buffering in cat is messing with the inherent line-buffering of a fifo. Maybe add an input redirect after each done rather than use a pipe (UUOC). Also put different sleeps and echoes in each loop to make the activities slow and visible. Feb 7, 2022 at 16:50

1 Answer 1

4

What's important to realise is that opening in read mode (not reading from) a fifo blocks blocks until some other process also opens it in write mode (and vice versa), and that as soon as that happens, a pipe is instantiated.

Then, more processes can hook onto that pipe by opening the fifo while that pipe is live.

That pipe is then destroyed as soon as there's no fd left open to it from any process, after which we're back to square one where another pipe can be instantiated as soon as the fifo is opened again for both reading and writing.

In:

[0]
[1] cat todo.pipe |
    [2] while read line && echo hej $line; do :; done \
            > output.pipe &
# The below 3 lines is all that's changed
[3] cat todo.pipe |
    [4] while read line && echo hej $line; do :; done \
            > output.pipe &

[5] echo "adam\n bertil\n carl" > todo.pipe &

[6] cat < output.pipe

The main shell process will spawn 4 processes at the same time, each living their own life independently and in parallel, the first one to run the first pipeline, second for second pipeline, third for echo, fourth for cat (after having opened output.pipe).

The pipeline ones will also spawn an extra process to run cat todo.pipe while the original process will carry on with the while loop concurrently.

So you'll have 6 (7 if you count the main shell process waiting for the last cat) started mostly concurrently. I've marked them with [1]..[6] above.

How they will be scheduled is up to the system's process scheduler. Running an external command like cat takes time, the actions that are performed by the shell itself will likely happen first.

All of 2, 4, 5, 6 start by opening a fifo file in the shell. 2 and 4 open output.pipe for writing and 6 for reading. Those will soon unlock each other and a pipe will be instantiated.

5 will hang on its write-only open of todo.pipe waiting for at least one of the cat processes to open it read-only.

Then 1 and 3 will race for that. Running cat involves executing /bin/cat which involves wiping the process memory, load the executable from disk, shared libraries, dynamic links, perform dynamic linking, and eventually run the code in it, where cat will parse its command line and eventually open that fifo file.

As soon as one of 1, 3 opens the fifo (let's say it's 1 here), 5 is unblocked. 1 will carry on doing a read() on that fd which will hang as there's nothing in the pipe yet.

5 is probably the one process that is going to be scheduled at that point. That's running a builtin echo command of the shell, so it will just do a write("adam...) and terminate, which involves closing its fd to output.pipe.

Then 1's read() can carry on now, cat reads by large chunks and will gobble up that whole small output in one read() and finish which involves closing its fd to the writing end of the pipe.

If 3 hadn't opened the fifo yet by that point, then the pipe is destroyed, and when 3 eventually opens the fifo, it will hang until something else opens the fifo in write mode and instantiates a new unrelated pipe which is not going to happen here.

If it weren't for the fact that the output.pipe were opened first, you could have the same problem with that fifo as well.

Now, even if you make it:

{
   cat | while...done &
   cat | while...done
} < todo.pipe > output.pipe &
echo ... > todo.pipe &
cat < output.pipe

Where todo.pipe is only opened once for reading so both cat's share the fd (same for output.pipe), avoiding this kind of problem, that probably won't be very useful.

The cat that does the first read() will gobble up the whole of echo's output, leaving nothing for the other. Any even if you replaced that echo with some something that outputs something larger than cat's reading buffer giving a chance to both cats to grab some piece each, each will end up with pieces cut in what appears like a random fashion.

If you remove the cat | to let the reads read directly from the pipe, it will be even worse as the read builtin reads one byte at a time, so you'll like end up with two competing reads reading one byte each in turn.

The only way for that approach to work would be by using cat and making sure the process feeding the todo.pipe does it slowly enough that the first task has already been read by one of the cat before the next one is fed in, feeding one todo task at a time with one write() system call, tasks not being larger than 4KiB nor than cat's read buffer size.

Better would be to have one process reading the pipe and dispatch the tasks to workers, for instance using things like GNU xargs -P or GNU parallel.

2
  • Thanks for your in-depth explanation. I guess one can summarise and say that FIFOs are not like queues in go or java, and are not useful for concurrency?
    – rutchkiwi
    Feb 9, 2022 at 9:37
  • @rutchkiwi they are like queues but there's no message boundary provided by the pipe API other than the byte. If your todo actions can be expressed in a fixed number of bytes (smaller than PIPEBUF (4K generally) to guarantee read/write atomicity), then you can safely have more than one concurrent reader provided the feeder does one write() per action and readers do read() of the full action size. Feb 9, 2022 at 11:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .