4

I'm not sure how to explain the problem in general, so I'll just use this example:

#!/bin/bash

cleanup() {
    rm "$myfifo"
    rm "$mylock"
    kill '$(jobs -p)'
}

writer() {
    for i in $(seq 0 100); do
        echo "$(date -R) writing \"$i\"."
        echo "$i" > "$myfifo"
    done
}

reader() {
    while true; do
        flock 3
        read -st 1 line
        status=$?
        if [ $status -eq 0 ]; then
            echo "$(date -R) reading \"$line\" in thread $1."
        else
            echo "$(date -R) status $status in thread $1.
            break
        fi
        flock -u 3
        sleep 10
    done 3<"$mylock" <"$myfifo"
}

trap cleanup EXIT

myfifo="$(mktemp)"
mylock="$(mktemp)"

rm "$myfifo"
mkfifo "$myfifo"

writer &

for i in $(seq 1 10); do
    reader $i &
    sleep 1
done

wait

Now I would expect the reading threads to each take a line (or a few lines) but the first reading process will take all the lines (in a random order which I don't understand but that's ok), put it in a buffer somewhere and all the other reading processes will not get any line.

Also the timeout parameter supplied to the read command doesn't seem to work because the readers 2-10 do not exit.

  1. Why?
  2. How can I fix this so the lines get (somewhat) evenly distributed among the readers?
  • And I then tried to manually write something into the pipe, the result: the hanging read process terminates (status 0) and the next one hangs. – Max Matti Mar 20 '17 at 0:12
7

Letting read timeout

read timeout actually works. The problem here is that opening a FIFO in reading mode blocks until the FIFO is opened in writing mode. And in this case, this is not read that is blocked, this is bash, when redirecting your FIFO to stdin.

Once some other process opens the FIFO for write, bash will successfully open the FIFO for read and will execute the read command (which will timeout as expected).

If you are using Linux, the man page for fifo tells us that "opening a FIFO for read and write will succeed both in blocking and nonblocking mode". Therefore, the following command will timeout even when no other process opens the FIFO for write:

read -st 1 data <> "$fifo"

Beware of the race condition

Once your shell process opens the FIFO for read, the writer(s) will then be unlocked and, by the time bash redirects the FIFO to stdin and calls read, the writer may be able to open the FIFO and write into it several times. Since you read only one line at a time, any line remaining to be read while the FIFO is closed at both ends will be lost. A better solution would be to keep the FIFO open by redirecting it to stdin for the whole while...done loop, as you did for fd 3. Something like:

while ...; do
    ...
    read -st 1 data
    ...
done 3<"$lock" < "$fifo"

Or even at an upper level, if you have several readers in parallel. What matters is to keep the FIFO open. Same for the writer side.

For example, with the code you posted with your update, the upper level would be:

# Writer
writer > "$myfifo" &

# Reader
for i in $(seq 1 10); do
    reader $i &
    sleep 1
done < "$myfifo"

Of course, remove the redirections to/from $myfifo everywhere else in your code, and remove the echo "$(date -R) writing \"$i\"." in your writer, or redirect it to stderr, else it would go to the FIFO.

  • But there's content which I put into the pipe (in another thread) which I haven't taken out yet. Is that lost? How can I make sure that don't just lose random content when reading from the pipe? – Max Matti Mar 20 '17 at 0:36
  • I you succeeded in writing to the FIFO, that means that another process had opened it in reading mode. If that another process only read one line of data and then closed the FIFO (and if there is no other reader at that time), the following lines that were written to the FIFO are lost. As I understand your program, the writer must write one line at a time (open, write, close) into the FIFO. – xhienne Mar 20 '17 at 0:48
  • The part of the bash script that writes into the pipe is a for loop that writes each line with one call of echo "stuff" >"$pipe". Should I implement some kind of waiting mechanism? – Max Matti Mar 20 '17 at 0:59
  • 2
    Theoretically, there is still a possibility that your for loop runs so fast that it executes two loops while there is only one read at the other end. – xhienne Mar 20 '17 at 1:04
  • That was exactly the issue. Thanks for the hint. Now I have to pipes (one in the other directionm basically working as a "hey, give me another line"). – Max Matti Mar 20 '17 at 1:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.