I am compressing a huge directory which could be more than 10 GB, It takes time to compress it. But at the same time, I have to download this to another new server.

So while the compressing is going on, Can I download this file from another server ? After downloading will it work properly ?

  • Don't think it will work. Maybe you can split the archive to few files and downloading the ready ones. – Svetlin Tonchev Dec 11 '15 at 12:22
  • Do you mean you want to download the uncompressed file from machineA to machineB and compress it while downloading? Or do you want to compress the directory on machineA and have the uncompressed directory copied to machineB? Please edit your question and clarify. – terdon Dec 11 '15 at 12:39

You can do all of that in two commands, one on server, one on client using tar, gzip, tee, nc.

On server : tar cf - dirname/ | gzip -c - |tee dirname.tar.gz | nc -l 3333

On the client nc serveraddress 3333 > dirname.tar.gz

nc stands for netcat, a usefull tool behaving like cat for network usage. nc -l 3333 will create a listening socket on local port 3333 and will stream the content of stdin to who ever connects to it.

nc serveraddress 3333 will establish a tcp connexion to server on port 3333, and stream in stdout everything it will receive in the socket connexion.

This is a very easy way to transfer file over different machines while performing various command chaining like tar/gzip/tee ....

  • I think, It might not work. IMO, the tar.gz is compressed file and once the complete compression is done, the headers might change. or In my opinion, the beginning of the file might changes after the compression is fully completed. i.e it could change some META data. – Mani Dec 12 '15 at 6:01
  • It's not a matter of Opinion. Did you try it or not ? and did it work or not ? – netmonk Dec 12 '15 at 13:34

Not sure if you can download an existing compress process but you might be able to use rsync with the --inplace directive. You will need to do this repetitively untill the host has completed the compression process. You will need to perform it once more after the compression is completed to get the final delta of changes.

Another way is to start downloading the file directly while creating the archive using:

ssh user@host "tar cf - /path/to/files|gzip -c -| \
tee /path/to/compressed_file" > /path/to/local_compressed_file

The above command will create a tar archive to stdout, compresses the stream also to stdout and using tee the file is written on the host as well as to stdout again which can be saved to a local copy.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.