`tail -f` consumes last line partially, doesn't care about newlines or nul

Question

I've got a temporary file (not a fifo/pipe) that needs to be watched by multiple reader scripts. Each script uses a background process to watch the temporary file, using this code:

function file_relay {
    # $1 is a regular file to read from
    local bg_file
    bg_file="$1"

    # $2 is a fifo to relay to
    local outfile
    outfile="$2"

    tail -f "$bg_file" | while read -r line
    do
        [[ ! -z "$line" ]] && { printf "%s" "$line" >>"$outfile"; }
    done
}

They need to read the entire file when they start, then watch for new lines, which the above function does:

file_relay /tmp/examplefile /tmp/examplefifo &

Each script will output lines to this file as well. So it's a multiple-writer and multiple-reader situation.

The problem is that sometimes tail -f doesn't wait for a full line to be available, even though I'm using printf to redirect to the file and I've got newlines at the end of the strings. This causes the lines read to be corrupted, with the first word of the last line being appended at the end of the previous one, so I get:

This is one lineThis

instead of

This is one line
This is another line

I've tried to work around printf's buffering, tail -f's buffering as well as use sync around writes to the file (the file is read only in the function above and I don't know how to force tail to run sync before attempting to read the entire line). stdbuf doesn't seem to have any effect anywhere, nor does using -z for tail, or terminating strings with $'\0' or anything else. The only thing that prevents it from happening right away is a single sync before the while loop starts, but that doesn't prevent it from happening after that.

Is there any way to force tail -f to read only complete lines?

Answer 1

short: not directly

long: it's not portable (not in POSIX), but if you limit your interest to Linux, you can pipe the output of tail -f through something which is line-buffered. For instance, as suggested in unix command 'tail' lost option '--line-buffered', GNU grep has a --line-buffered option, allowing you to do this

tail -f "$bg_file" | grep --line-buffered -E '^.*$'

However as the manual points out

--line-buffered
Use line buffering, it can be a performance penality.

(FreeBSD has the same option and comment from OpenBSD in 2004, not POSIX yet...).

The documentation doesn't point this out, but the initial commit in 2001 had in mind the time spent doing fflush.