Why does the following command not produce any output?

$ tail -f /etc/passwd | tail

After reading about buffering, I tried the following to no avail:

$ tail -f /etc/passwd | stdbuf -oL tail

Note that the following does produce output:

$ tail /etc/passwd | tail

So does this:

$ tail -f /etc/passwd | head

I am using tail version 8.21 (GNU coreutils).

  • 15
    What are the last 10 digits of π? – Keith Thompson Sep 14 '14 at 9:56
up vote 13 down vote accepted

I thought I had seen everything in UNIX. This question slapped me out of my smugness. What a great question!

tail shows the last X lines. tail -f does the same, but essentially in an infinite loop: on start-up, show the last X lines of the file, then using some OS magic (like inotify), monitor and show new lines.

To do its job, tail must be able to locate the end of the file. If tail cannot find the file's end, it cannot show the last X lines, because "last" is undefined. So what does tail do in this case? It waits until it does find the end of the file.

Consider this:

$ chatter() { while :; do date; sleep 1; done; }
$ chatter | tail -f

This never appears to make progress, because there is never a definite end of file from chatter.

You get the same behavior if you ask tail to give you the last lines from a file system pipe. Consider:

$ mkfifo test.pipe
$ tail test.pipe

stdbuf to get around the perceived problem was a noble attempt. The key fact though is that I/O buffering isn't the root cause: the lack of a definite end-of-file is. If you check out the tail.c source code, you'll see the file_lines function comment reads:

END_POS is the file offset of EOF (one larger than offset of last byte).

and that's the magic. You need an end-of-file for tail to work in any configuration. head doesn't have that restriction, it just needs a start of file (which it might not have, try head test.pipe). Stream oriented tools like sed and awk need neither a start or end of file: they work on buffers.

tail -f's tail is actually something unknown in the present, so how should the next tail know it. On the other hand tail -f's head is something already known and could therefor be processed.

Or to put it simpler: tail is relative to the end of the file, but the output stream of tail -f got no EOF (at least not before its termination).

If you find the first tail's pid and kill it, you should then see the output from the second.

Technical answer

When running with a stream as input, tail keeps an n-line buffer that it fills as it reads the stream, but it cannot output those lines until it reaches the end of the stream, i.e. it receives a special EOF code when trying to read from the input stream. The invocation tail -f does not exit, thus it will never close its stream, which makes it impossible to e.g. return the 10 last lines of that stream.

The function of tail is to show the last part - "tail" - of the input or file. (The option -f is about what it does later, so that is not relevant here.)

Let's think about a file:

What is the last part of a file?
Let's say it's the last n lines of a file.

When we read line i of the input file, how to decide it needs to be printed or not?
We do not know whether it is in the last part - because we do not know which the last line will be. So we can not print it now.

We need to keep the line until it becomes clear that is's part of the last n lines, or can no longer be part of it, because we know n further lines

If we now come to the end of the file, we know that the last n lines we kept are in fact the last n lines of the file.

Now, in the case of

tail -f /etc/passwd | tail

the first tail reads the file, and then waits to get more data from it, to write that out too. So it will not signal an end of file to the second tail when it comes to the end of the file it reads. Without that, the second tail never gets notified of the end of file, so it can never find out which are the last lines it should print.

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.