"tail -f", but on a file which gets rewritten (downloaded) again and again without outputting then content over and over again?

Question

I've got log files which get downloaded via cron job. If a file is updated on the remote location, the local copy gets rewritten from the beginning even if only data has been appended.

Tools like tail -f or since seem to see this as "the file has been replaced" and start outputting them from the beginning again. i.e. repeat all already known content. Especially those two tools explicitly mention this on STDERR.

So if I call this in one terminal:

for j in $(seq 2 4) ; do for i in $(seq 1 $j) ; do echo $i ; sleep 1; done > /tmp/foo; done

I get these warnings with both, tail -f /tmp/foo and tail -F /tmp/foo in another terminal:

1
2
tail: /tmp/foo: file truncated
1
2
3
tail: /tmp/foo: file truncated
1
2
3
4

And with while sleep 0.25; do since /tmp/foo; done, I get one of these error messages:

1
2
since: considering /tmp/foo to be truncated, displaying from start
1
2
3
since: considering /tmp/foo to be truncated, displaying from start
1
2
3
4

since explicitly uses the inode and not the file name as key for a file. tail probably does something similar to recognise truncated files.

Another "tool" I tried is the Perl library File::Tail, but it has the same "issue", just without warnings (at least with default settings).

So I wonder: Is a way or tool which does not look at the inode but just at the contents of the file and only restarts if data has not just been appended?

What I would like to have is just this output:

1
2
3
4

(I've seen tail -f, but when the file is deleted and re-created (not appended), but it does not help as it still restarts outputting the file's content from the beginning as well.)

And yes, I'm aware that such a tool either needs a cache of all seen (but not truncated) data or at least hashsums of it.

Answer 1

That's tail trying to be too smart for your use case. Here, you could do:

{
  tail
  while cat; do
    sleep 1
  done
} < /tmp/foo

Same as what the original implementation of tail did (when -f was added in SysIII in 1980).

Better with a shell where cat and sleep are builtin or where there are equivalent builtin commands. Or do it in perl/python...

For instance, in ksh93, sleep is builtin by default and the cat builtin can be enabled with builtin cat. In zsh, you'd use sysread in a loop instead of cat and zselect in place of sleep:

zmodload zsh/zselect
zmodload zsh/system
readall() while sysread -s 65536 -o1; do continue; done

{
  tail
  while readall; do
    zselect -t 100
  done
} < /tmp/foo

If a new file was being created each time, tail -f would not detect it. Still if that was the case, you could still do it by doing something like:

#! /bin/zsh -
zmodload zsh/zselect
zmodload zsh/system
readall() while sysread -s 65536 -o1; do continue; done

file=${1-/tmp/foo}
{
  tail
  (( offset = systell(0) ))
} < $file || exit
while true; do
  if
    sysopen -ru0 -- $file 2> /dev/null &&
      sysseek $offset &&
      readall
  then
    (( offset = systell(0) ))
  fi
  zselect -t 100
done

Where we reopen the file and seek to the last known offset at each iteration.