6

I'm using the following statement (simplified version):

tail -f -c+1 <filename>

in order to pipe a file to a process.

What I've found, though, is that there is a number of lines at the end which are not piped.

A specific example is piping a mysql file, and stopping when the end is reached:

tail -f -c+1 mysqdump.sql | sed '/^-- Dump completed/ q0'

This doesn't work - the last line of the dump -- Dump completed [...] is not piped to sed.

My guess is that the tail -f buffer, in this case, is flushed only when it's full.

Does anybody know how can I workaround this?

=================

I've found the cause - the description is not complete (and the code doesn't exhibit the behavior).

The problem happens when piping from a compressed (lzma) file:

tail -f -c+1 <filename.lzma> | lzma -d | sed '/^-- Dump completed/ q0'

Most likely, tail is not sending the last compressed block, because it doesn't detect any new line, as the input is binary.

9

tail -f flushes after every input line. You can confirm this with strace (or truss or whatever your unix variant provides to trace processes' system calls).

If there is an incomplete line, tail -f keeps waiting for the next newline. It's a tool designed for text files. If you need to tail a binary file (e.g. if -- Dump completed is not followed by a newline), you'll have to use a custom tool.

If you've redirected sed's output away from the terminal, it will be doing its own buffering. Try stdbuf or unbuffer.

  • I was in fact executing on a binary file being remotely copied (I purposely don't want to use "ssh cat"). I worked the problem around using scp on a named pipe. – Marcus Aug 31 '12 at 8:41
  • For sed you can use -u if you don't want buffer – agate Jul 10 '18 at 16:59
2

Example using stdbuf -o0. Reference: https://superuser.com/a/1123674/126847

tail -F cpu.log | stdbuf -o0 tr '=' ' ' | stdbuf -o0 awk '$6 > 1'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.