46

A file is being sequentially downloaded by wget.

If I start unpacking it with cat myfile.tar.bz2 | tar -xj, it may unpack correctly or fail with "Unexpected EOF", depending on what is faster.

How to "cat and follow" a file, i.e. output content of the file to stdout, but don't exit on EOF, instead keep subsribed to that file and continue outputting new portions of the data, exiting only if the file is closed by writer and not re-opened within N seconds.


I've created a script cat_and_follow based on @arielCo's answer that also terminates the tail when the file is not being opened for writing anymore.

9
  • 2
    I think you're looking for the tail command with its follow option.
    – keshlam
    Jun 30, 2014 at 2:47
  • tail works with binary files as well?
    – Vi.
    Jun 30, 2014 at 8:43
  • 2
    You might also like less +F...
    – mikeserv
    Jun 30, 2014 at 11:27
  • 1
    @JosephR., 1. To save up time by parallelizing; 2. To even up system load (avoiding/minimizing resource-intensive full throttle unpacking phase); 3. To avoid remembering about the "unpack" command later (or scheduling it) or typing chained command in the first place (when I'm not yet sure that I want to unpack it).
    – Vi.
    Jun 30, 2014 at 15:18
  • 2
    I edited your question to remove the "Answer" section. As a general rule, answering your own question is fine and encouraged but since it was based on the accepted answer I just changed your wording to clarify.
    – terdon
    Jun 30, 2014 at 16:18

4 Answers 4

32
tail +1f file

I tested it on Ubuntu with the LibreOffice source tarball while wget was downloading it:

tail +1f libreoffice-4.2.5.2.tar.xz | tar -tvJf -

It also works on Solaris 10, RHEL3, AIX 5 and Busybox 1.22.1 in my Android phone (use tail +1 -f file with Busybox).

11
  • 1
    Will it automatically exit when downloading is finished? I expect that "follower" to scan /proc/*/fd/ periodically and check whether the file is closed for good.
    – Vi.
    Jun 30, 2014 at 8:45
  • 3
    I didn't expect that tail works with binray files...
    – Vi.
    Jun 30, 2014 at 8:47
  • Created a script that does scanning /proc/../fd and auto-termination.
    – Vi.
    Jun 30, 2014 at 9:11
  • 1
    You don't need GNU tail, I believe the -f option is standard. However, it won't terminate automatically when the download is finished.
    – Barmar
    Jul 2, 2014 at 19:31
  • 5
    In Ubuntu tail +1f does not work. You have to do tail -n +1
    – blueFast
    Feb 18, 2015 at 17:57
12

The problem is that cat is not aware that the file is still being appended. As soon as cat encounters the (current) end of the file it exits.

You have to make wget write to a pipe (or FIFO) in order to avoid this.

wget -O - http://... | tar -xjf -
3
  • 7
    Or curl, which outputs to stdout by default. And (1) -f - is redundant, tar reads from stdin by default, and (2) most tars can detect the compression automatically, so the j is often unnecessary. curl http://... | tar x
    – Kevin
    Jun 30, 2014 at 3:17
  • 2
    This will download without saving the unpacked file. Also it will hinder continuing the download in case of bad network.
    – Vi.
    Jun 30, 2014 at 8:44
  • 4
    @Vi. You can save the file by using tee like this: curl http://… | tee ….tbz | tar -xj, but resuming the download gets more complicated than just invoking the same command again.
    – kasperd
    Jun 30, 2014 at 11:40
12

To read and follow a file from the beginning until interrupted:

tail -fn +1 file

To demonstrate that, try this (assuming Bash with GNU Coreutils):

(while true; do printf . >> /tmp/file; sleep 1; done)&
tail -fn +1 /tmp/file  # (Ctrl-C to interrupt, of course, or otherwise kill it.)
kill %  # Kills the while-loop.

(Note: The +1f mentioned by others is interpreted as a filename, at least in the GNU tail command.)

The above works for a single file. Concatenation of multiple files would not be able to follow all of them deterministically, without hanging on the first. To ‘cat and follow’, following only the last file, one can use process substitution. Here's another demonstration:

printf file1 > /tmp/file1; printf file2 > /tmp/file2
(while true; do printf . | tee -a /tmp/file{1,2} > /dev/null; sleep 1; done)&
cat /tmp/file1 <(tail -fn +1 /tmp/file2)  # (Interrupt or kill it.)
kill %  # Kills the while-loop.
2
  • This worked for me while tail +1f file didn't.
    – 2xsaiko
    Jul 29, 2017 at 19:36
  • This answer is much better than tail +1f when combining it with other options such as --pid Mar 15, 2018 at 19:40
0

With less +F [My favorite]

Tested on Linux Ubuntu 18.04 with GNU less version 487, as shown by less --version.

You can also use less:

less -N +F path/to/some/growing/log_file.log

You may also want to follow the name of a file instead of its file descriptor by adding the --follow-name option:

# [My favorite command overall]
less -N --follow-name +F path/to/some/growing/log_file.log

This is useful to follow the file named log_file.log, for instance, even when a rotating log system renames this file to log_file.log.1 as it rotates to a new log file to begin logging. WithOUT --follow_name, less would continue to follow the file log_file.log.1, which is now frozen and not growing, whereas WITH --follow-name, less, will see the name changed and automatically open and begin following the new log_file.log file. See here.

The -N shows line numbers. The + causes less to run the command right after the + symbol when it opens. The F command causes it to continually read and load (ie: "follow") the end of the file, which is particularly useful to see growing log files grow. Pressing F (ie: Shift + F) while less is open is identical to pressing Ctrl + End.

To interrupt and stop this continual loading effect while less is running, press Ctrl + C. Now you can go back to being able to use less like normal, scrolling up and down to view data as you desire. Then press q to exit, like normal. Or, you can resume following the file by typing F (Shift + F).

Note: to just open in less and jump to the end of the file but NOT continually load ("follow") the new contents as they are added, use the G command (Shift + G in less if less is already running) instead of F (Shift + F if less is already running):

less -N +G path/to/some/growing/log_file.log

With tail -f [Best option on BusyBox]

Note that if using tail on a regular Linux machine, not an embedded Linux machine with a BusyBox implementation of tail, you can use tail's --follow=name option to do the equivalent of less's --follow-name option described above.

The following was tested with BusyBox v1.31.1, as shown by busybox --help.

The less +F option isn't available on embedded Linux systems running busybox, so for these systems use tail -f instead:

# Just show new contents as they come into the file
tail -f path/to/some/growing/log_file.log

# Also print the entire file first, starting at the first line (`+1`), before
# following and loading new contents continually
tail +1 -f path/to/some/growing/log_file.log

It's not quite as convenient as less, but it still works just fine. Press Ctrl + C to kill the output. Then you can scroll up in the terminal to see previous lines in case you need to "pause" the output and view something more-closely. To "resume" viewing the file, press the Up Arrow key of course to recall your previous command so you can easily run the same tail -f command again.

Some other useful options on BusyBox are -s SECONDS and possibly -F. Ex:

# only check and load new contents at the end of the file every 2 seconds
tail -f -s 2 path/to/some/growing/log_file.log

Here is the full help menu:

# tail --help
BusyBox v1.31.1 (2021-11-20 02:33:23 UTC) multi-call binary.

Usage: tail [OPTIONS] [FILE]...

Print last 10 lines of each FILE (or stdin) to stdout.
With more than one FILE, precede each with a filename header.

    -f      Print data as file grows
    -c [+]N[kbm]    Print last N bytes
    -n N[kbm]   Print last N lines
    -n +N[kbm]  Start on Nth line and print the rest
    -q      Never print headers
    -s SECONDS  Wait SECONDS between reads with -f
    -v      Always print headers
    -F      Same as -f, but keep retrying

N may be suffixed by k (x1024), b (x512), or m (x1024^2).

With watch [also works on BusyBox]

Here is another option. This also works fine in BusyBox:

# Continually view the last 20 messages of the log file every 1 second
watch -n 1 'tail -n 20 path/to/some/growing/log_file.log'

References

  1. I first learned about less +F here: Open `less` scrolled to the end
  2. I first learned about less +G here: Open `less` scrolled to the end
  3. Where I learned about less's --follow-name option: https://unix.stackexchange.com/a/196349/114401

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .