5 added 11 characters in body
source | link

I'm not sure how to explain the problem in general, so I'll just use this example:

#!/bin/bash

cleanup() {
    rm "$myfifo"
    rm "$mylock"
    kill '$(jobs -p)'
}

writer() {
    for i in $(seq 0 100); do
        echo "$(date -R) writing \"$i\"."
        echo "$i" > "$myfifo"
    done
}

reader() {
    while true; do
        flock 3
        read -st 1 line
        status=$?
        if [ $status -eq 0 ]; then
            echo "$(date -R) reading \"$line\" in thread $1."
        else
            echo "$(date -R) status $status in thread $1.
            break
        fi
        flock -u 3
        sleep 10
    done 3<"$mylock" <"$myfifo"
}

trap cleanup EXIT

myfifo="$(mktemp)"
mylock="$(mktemp)"

rm "$myfifo"
mkfifo "$myfifo"

writer &

for i in $(seq 1 10); do
    reader $i &
    sleep 1
done

wait

Now I would expect the reading threads to each take a line (or a few lines) but the first reading process will take all the lines (in a random order which I don't understand but that's ok), put it in a buffer somewhere and all the other reading processes will not get any line.

Also the timeout parameter supplied to the read command doesn't seem to work because the readers 2-10 do not exit.

  1. Why?
  2. How can I fix this so the lines get (somewhat) evenly distributed among the readers?

I'm not sure how to explain the problem in general, so I'll just use this example:

#!/bin/bash

cleanup() {
    rm "$myfifo"
    rm "$mylock"
    kill '$(jobs -p)'
}

writer() {
    for i in $(seq 0 100); do
        echo "$(date -R) writing \"$i\"."
        echo "$i" > "$myfifo"
    done
}

reader() {
    while true; do
        flock 3
        read -st 1 line
        status=$?
        if [ $status -eq 0 ]; then
            echo "$(date -R) reading \"$line\" in thread $1."
        else
            echo "$(date -R) status $status in thread $1.
        fi
        flock -u 3
        sleep 10
    done 3<"$mylock" <"$myfifo"
}

trap cleanup EXIT

myfifo="$(mktemp)"
mylock="$(mktemp)"

rm "$myfifo"
mkfifo "$myfifo"

writer &

for i in $(seq 1 10); do
    reader $i &
    sleep 1
done

wait

Now I would expect the reading threads to each take a line (or a few lines) but the first reading process will take all the lines (in a random order which I don't understand but that's ok), put it in a buffer somewhere and all the other reading processes will not get any line.

Also the timeout parameter supplied to the read command doesn't seem to work because the readers 2-10 do not exit.

  1. Why?
  2. How can I fix this so the lines get (somewhat) evenly distributed among the readers?

I'm not sure how to explain the problem in general, so I'll just use this example:

#!/bin/bash

cleanup() {
    rm "$myfifo"
    rm "$mylock"
    kill '$(jobs -p)'
}

writer() {
    for i in $(seq 0 100); do
        echo "$(date -R) writing \"$i\"."
        echo "$i" > "$myfifo"
    done
}

reader() {
    while true; do
        flock 3
        read -st 1 line
        status=$?
        if [ $status -eq 0 ]; then
            echo "$(date -R) reading \"$line\" in thread $1."
        else
            echo "$(date -R) status $status in thread $1.
            break
        fi
        flock -u 3
        sleep 10
    done 3<"$mylock" <"$myfifo"
}

trap cleanup EXIT

myfifo="$(mktemp)"
mylock="$(mktemp)"

rm "$myfifo"
mkfifo "$myfifo"

writer &

for i in $(seq 1 10); do
    reader $i &
    sleep 1
done

wait

Now I would expect the reading threads to each take a line (or a few lines) but the first reading process will take all the lines (in a random order which I don't understand but that's ok), put it in a buffer somewhere and all the other reading processes will not get any line.

Also the timeout parameter supplied to the read command doesn't seem to work because the readers 2-10 do not exit.

  1. Why?
  2. How can I fix this so the lines get (somewhat) evenly distributed among the readers?
4 Cleaner output.
source | link

My bash-script contains the following lines:

#!/bin/bash

cleanup() {
    rm "$myfifo"
    rm "$mylock"
    kill '$(jobs -p)'
}

writer() {
    for i in $(seq 0 100); do
        echo "test1""$(date 1>&2-R) writing \"$i\"."
        echo "$i" > "$myfifo"
    done
}

reader() {
    while true; do
        flock 3
echo "test2" 1>&2
      read -st 1 data <"$fifo"line
echo "test3" 1>&2
read_status=$      status=$?
echo "test4" 1>&2
flock      if [ $status -ueq 30 ]; then
            echo "Status"$(date $read_status-R) forreading $data\"$line\" in thread $1." 
 1>&2       else

which are inside a function that gets executed a few times in parallel by using & at the end.

Now I expect the amount of "test4" in my output to be equal to the amount of "test1" in my output because no matter the current status of the pipe, read either gets 1 line from the pipe and returns or the timeout hits and it returns.

fd 3 was assigned using 3<"$lock" after the done operator of the current while-block, $fifo is an existing named pipe which was created using mkfifo.

However it the output looks like this:

[..            echo "$(date -R) status $status in thread $1.]
test1
test2
test3
test4
Status 0 for [...]     fi
[...]        flock -u 3
test1        sleep 10
test2    done 3<"$mylock" <"$myfifo"
test3}
test4
Status 0trap forcleanup [...]
[...]EXIT
test1
test2myfifo="$(mktemp)"
test3mylock="$(mktemp)"
test4
Status 0rm for"$myfifo"
mkfifo [...]"$myfifo"
[...]
test1writer &
test2
[...]for i in $(seq 1 10); do
test1    reader $i &
[...]    sleep 1
test1done
[...]
test1wait

And then comes nothing forNow I would expect the reading threads to each take a long timeline (or a few lines) but the first reading process will take all the lines (in a random order which I don't understand but that's ok), put it in a buffer somewhere and all the other reading processes will not get any line.

Also the timeout parameter supplied to the read command doesn't seem to work because the readers 2-10 do not exit.

  1. Why?
  2. How can I fix this so the lines get (somewhat) evenly distributed among the readers?

My bash-script contains the following lines:

echo "test1" 1>&2
flock 3
echo "test2" 1>&2
read -st 1 data <"$fifo"
echo "test3" 1>&2
read_status=$?
echo "test4" 1>&2
flock -u 3
echo "Status $read_status for $data." 1>&2

which are inside a function that gets executed a few times in parallel by using & at the end.

Now I expect the amount of "test4" in my output to be equal to the amount of "test1" in my output because no matter the current status of the pipe, read either gets 1 line from the pipe and returns or the timeout hits and it returns.

fd 3 was assigned using 3<"$lock" after the done operator of the current while-block, $fifo is an existing named pipe which was created using mkfifo.

However it the output looks like this:

[...]
test1
test2
test3
test4
Status 0 for [...]
[...]
test1
test2
test3
test4
Status 0 for [...]
[...]
test1
test2
test3
test4
Status 0 for [...]
[...]
test1
test2
[...]
test1
[...]
test1
[...]
test1

And then comes nothing for a long time.

#!/bin/bash

cleanup() {
    rm "$myfifo"
    rm "$mylock"
    kill '$(jobs -p)'
}

writer() {
    for i in $(seq 0 100); do
        echo "$(date -R) writing \"$i\"."
        echo "$i" > "$myfifo"
    done
}

reader() {
    while true; do
        flock 3
        read -st 1 line
        status=$?
        if [ $status -eq 0 ]; then
            echo "$(date -R) reading \"$line\" in thread $1." 
        else
            echo "$(date -R) status $status in thread $1.
        fi
        flock -u 3
        sleep 10
    done 3<"$mylock" <"$myfifo"
}

trap cleanup EXIT

myfifo="$(mktemp)"
mylock="$(mktemp)"

rm "$myfifo"
mkfifo "$myfifo"

writer &

for i in $(seq 1 10); do
    reader $i &
    sleep 1
done

wait

Now I would expect the reading threads to each take a line (or a few lines) but the first reading process will take all the lines (in a random order which I don't understand but that's ok), put it in a buffer somewhere and all the other reading processes will not get any line.

Also the timeout parameter supplied to the read command doesn't seem to work because the readers 2-10 do not exit.

  1. Why?
  2. How can I fix this so the lines get (somewhat) evenly distributed among the readers?
3 Not related to flock
| link
    Tweeted twitter.com/StackUnix/status/843714336434667524
2 clarified one paragraph
source | link
1
source | link