Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

A file is being sequentially downloaded by wget.

If I start unpacking it with cat myfile.tar.bz2 | tar -xj, it may unpack correctly or fail with "Unexpected EOF", depending on what is faster.

How to "cat and follow" a file, i.e. output content of the file to stdout, but don't exit on EOF, instead keep subsribed to that file and continue outputting new portions of the data, exiting only if the file is closed by writer and not re-opened within N seconds.


I've created a script cat_and_follow based on @arielCo's answer that also terminates the tail when the file is not being opened for writing anymore.

share|improve this question
    
I think you're looking for the tail command with its follow option. –  keshlam Jun 30 at 2:47
    
tail works with binary files as well? –  Vi. Jun 30 at 8:43
1  
You might also like less +F... –  mikeserv Jun 30 at 11:27
    
I know you found your solution, but I have a question, why not simply do: wget ... && tar xjvf ... –  Joseph R. Jun 30 at 13:44
    
@JosephR., 1. To save up time by parallelizing; 2. To even up system load (avoiding/minimizing resource-intensive full throttle unpacking phase); 3. To avoid remembering about the "unpack" command later (or scheduling it) or typing chained command in the first place (when I'm not yet sure that I want to unpack it). –  Vi. Jun 30 at 15:18

2 Answers 2

up vote 14 down vote accepted
tail +1f file

I tested it on Ubuntu with the LibreOffice source tarball while wget was downloading it:

tail +1f libreoffice-4.2.5.2.tar.xz | tar -tvJf -

It also works on Solaris 10, RHEL3, AIX 5 and Busybox 1.22.1 in my Android phone (use tail +1 -f file with Busybox).

share|improve this answer
1  
Will it automatically exit when downloading is finished? I expect that "follower" to scan /proc/*/fd/ periodically and check whether the file is closed for good. –  Vi. Jun 30 at 8:45
1  
I didn't expect that tail works with binray files... –  Vi. Jun 30 at 8:47
    
Created a script that does scanning /proc/../fd and auto-termination. –  Vi. Jun 30 at 9:11
    
@Vi. It's great that you solved your problem. Can you share your solution? –  Schism Jun 30 at 13:29
1  
You don't need GNU tail, I believe the -f option is standard. However, it won't terminate automatically when the download is finished. –  Barmar Jul 2 at 19:31

The problem is that cat is not aware that the file is still being appended. As soon as cat encounters the (current) end of the file it exits.

You have to make wget write to a pipe (or FIFO) in order to avoid this.

wget -O - http://... | tar -xjf -
share|improve this answer
7  
Or curl, which outputs to stdout by default. And (1) -f - is redundant, tar reads from stdin by default, and (2) most tars can detect the compression automatically, so the j is often unnecessary. curl http://... | tar x –  Kevin Jun 30 at 3:17
2  
This will download without saving the unpacked file. Also it will hinder continuing the download in case of bad network. –  Vi. Jun 30 at 8:44
3  
@Vi. You can save the file by using tee like this: curl http://… | tee ….tbz | tar -xj, but resuming the download gets more complicated than just invoking the same command again. –  kasperd Jun 30 at 11:40

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.