A file is being sequentially downloaded by wget.

If I start unpacking it with cat myfile.tar.bz2 | tar -xj, it may unpack correctly or fail with "Unexpected EOF", depending on what is faster.

How to "cat and follow" a file, i.e. output content of the file to stdout, but don't exit on EOF, instead keep subsribed to that file and continue outputting new portions of the data, exiting only if the file is closed by writer and not re-opened within N seconds.

I've created a script cat_and_follow based on @arielCo's answer that also terminates the tail when the file is not being opened for writing anymore.

  • 2
    I think you're looking for the tail command with its follow option.
    – keshlam
    Jun 30 '14 at 2:47
  • tail works with binary files as well?
    – Vi.
    Jun 30 '14 at 8:43
  • 2
    You might also like less +F...
    – mikeserv
    Jun 30 '14 at 11:27
  • 1
    @JosephR., 1. To save up time by parallelizing; 2. To even up system load (avoiding/minimizing resource-intensive full throttle unpacking phase); 3. To avoid remembering about the "unpack" command later (or scheduling it) or typing chained command in the first place (when I'm not yet sure that I want to unpack it).
    – Vi.
    Jun 30 '14 at 15:18
  • 1
    I edited your question to remove the "Answer" section. As a general rule, answering your own question is fine and encouraged but since it was based on the accepted answer I just changed your wording to clarify.
    – terdon
    Jun 30 '14 at 16:18
tail +1f file

I tested it on Ubuntu with the LibreOffice source tarball while wget was downloading it:

tail +1f libreoffice- | tar -tvJf -

It also works on Solaris 10, RHEL3, AIX 5 and Busybox 1.22.1 in my Android phone (use tail +1 -f file with Busybox).

  • 1
    Will it automatically exit when downloading is finished? I expect that "follower" to scan /proc/*/fd/ periodically and check whether the file is closed for good.
    – Vi.
    Jun 30 '14 at 8:45
  • 2
    I didn't expect that tail works with binray files...
    – Vi.
    Jun 30 '14 at 8:47
  • Created a script that does scanning /proc/../fd and auto-termination.
    – Vi.
    Jun 30 '14 at 9:11
  • 1
    You don't need GNU tail, I believe the -f option is standard. However, it won't terminate automatically when the download is finished.
    – Barmar
    Jul 2 '14 at 19:31
  • 4
    In Ubuntu tail +1f does not work. You have to do tail -n +1
    – blueFast
    Feb 18 '15 at 17:57

The problem is that cat is not aware that the file is still being appended. As soon as cat encounters the (current) end of the file it exits.

You have to make wget write to a pipe (or FIFO) in order to avoid this.

wget -O - http://... | tar -xjf -
  • 7
    Or curl, which outputs to stdout by default. And (1) -f - is redundant, tar reads from stdin by default, and (2) most tars can detect the compression automatically, so the j is often unnecessary. curl http://... | tar x
    – Kevin
    Jun 30 '14 at 3:17
  • 2
    This will download without saving the unpacked file. Also it will hinder continuing the download in case of bad network.
    – Vi.
    Jun 30 '14 at 8:44
  • 3
    @Vi. You can save the file by using tee like this: curl http://… | tee ….tbz | tar -xj, but resuming the download gets more complicated than just invoking the same command again.
    – kasperd
    Jun 30 '14 at 11:40

To read and follow a file from the beginning until interrupted:

tail -fn +1 file

To demonstrate that, try this (assuming Bash with GNU Coreutils):

(while true; do printf . >> /tmp/file; sleep 1; done)&
tail -fn +1 /tmp/file  # (Ctrl-C to interrupt, of course, or otherwise kill it.)
kill %  # Kills the while-loop.

(Note: The +1f mentioned by others is interpreted as a filename, at least in the GNU tail command.)

The above works for a single file. Concatenation of multiple files would not be able to follow all of them deterministically, without hanging on the first. To ‘cat and follow’, following only the last file, one can use process substitution. Here's another demonstration:

printf file1 > /tmp/file1; printf file2 > /tmp/file2
(while true; do printf . | tee -a /tmp/file{1,2} > /dev/null; sleep 1; done)&
cat /tmp/file1 <(tail -fn +1 /tmp/file2)  # (Interrupt or kill it.)
kill %  # Kills the while-loop.
  • This worked for me while tail +1f file didn't.
    – 2xsaiko
    Jul 29 '17 at 19:36
  • This answer is much better than tail +1f when combining it with other options such as --pid Mar 15 '18 at 19:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.