3

I am new to these commands. I am trying to gzip a local folder and unzip the same on the remote server. The thing is, gzipping and unzip must happen on the fly. I tried many and one of the closest I believe is this:

tar cf dist.tar ~/Documents/projects/myproject/dist/ | ssh root@13.21.13.12:~/public_html/ "tar zx ~/Documents/projects/myproject/dist.tar"

As you can see above, I am trying to send out the dist folder to the remote server, but before that I am trying to compress the folder on the fly (looks like that is not happening in above command).

  • local folder: ~/Documents/projects/myproject/dist/
  • remote folder: ~/public_html (directly deploying to live)

Of course, the gzip created file must not be there, it should happen on the fly.

My intention is to run the above like through a file like sh file.command. In other words, I am trying to deploy my compiled project which is in dist folder, to live when the sh command is executed. I don't want to do this manually every time I make a change in my project.

6

If you have rsync then use that instead, as it makes use of existing files to allow it to transfer only differences (that is, parts of files that are different):

rsync -az ~/Documents/projects/myproject/dist/ root@13.21.13.12:public_html/

Add the --delete flag to to completely overwrite the target directory tree each time. If you want to see what's going on, add -v.

If you don't have rsync, then this less efficient solution using tar will suffice:

( cd ~/Documents/projects/myproject/dist && tar czf - . ) |
    ssh root@13.21.13.12 'cd public_html && tar xzf -'

Notice that the writing and reading of the compressed tarball is via stdout and stdin (the - filename). If you were using GNU tar you could use the -C to set a correct directory before processing, whereas here we've used old-fashioned (traditional?) cd. Add the v flag (on the receiving side) to see what's going on, i.e. tar xzvf ....

  • Thanks.. somehow tar didn't work me at all.. rsync worked as smooth as butter :D.. Did brew install rsync to install it before. – Mr_Green Jul 1 at 14:22
  • Isn't brew a Mac thing? – roaima Jul 1 at 14:23
  • yeah, I am on mac. btw, when I tried the same again it is ending up fast saying sending incremental file list.. is this fine? – Mr_Green Jul 1 at 14:25
  • yeah seems fine.. it is just taking the changed files and deploying instead of all files again. seems it is tracking the changes somehow. – Mr_Green Jul 1 at 14:30
  • 1
    I've simplified the tar command so that it should now work on a Mac. But rsync is still better. – roaima Jul 1 at 14:51
3

Instead of creating a file on disk, you need to send the output of tar to STDOUT, so it can then be sent over the wire using ssh:

$ tar -zcf - /path/to/files | ssh user@host "tar -zx - -C /path/to/destination"

Note the "-" characters where you are using file names. These tell tar on the sending side to write to STDOUT, and on the receiving side to read from STDIN. The archived data never gets written to the local disk.

You can't specify a path when logging in with ssh (at least, I've never figured out how to), so you need to tell tar on the receiving side where to write its output - that's what the -C /path/to/destination is for.

As currently written, the command would lead to files being written on the remote side nested in the local directories - you'd end up with

/path/to/files/index.html --> /path/to/destination/path/to/files/index.html

If, instead, you want /path/to/destination/index.html, say this:

$ tar -zcf - -C /path/to/files . | ssh user@host "tar -zx - -C /path/to/destination"
2

You are telling tar to write the tarball into the file dist.tar. With this command you are not sending the tarball to the remote machine.

You must make tar write its output to stdout using -f -. The hyphen means stdout. If you add a -z option, then tar will compress the tarball before writing it to stdout, which is a good idea in 99% of all use-cases where one sends a tarball to a remote machine.

On the remote side you extract with tar -x. In your example you have the -z option only on the remote side. This does not make sense. You must select the same compression method on both sides, i.e. use -z on both sides or don't use it on both sides.

This might work for you

tar -czf - ~/Documents/projects/myproject/dist/ | ssh root@13.21.13.12 "tar -C ~/public_html/ -xzf -"

(Disclaimer: I didn't try it out. It may contain errors.)

Since the remote tar is supposed to read the tarball from stdin, you don't provide a filename on this side too.

  • 1
    Thanks for the answer.. I am not sure whether this error.. I am seeing Removing leading '/' from member names. it just runs without any messages and when I check the remote.. I am seeing no change. – Mr_Green Jul 1 at 14:00
1

You are trying to accomplish two things:

  1. Copy the directory tree to the remote site
  2. Compress the data being transferred

Other answers have mentioned using rsync, which can accomplish both goals (giving a "-z" option to rsync enables compression).

tar is a good tool to accomplish goal #1, and with command line options (like the "z" flag given in other answers) it can also achieve goal #2.

The SSH connection can also itself be used to meet your goal of compression. SSH has a compression option that is off by default, but can be enabled at the command line or in the ssh_config file (see https://linux.die.net/man/5/ssh_config). SSH version 2 with compression enabled is equivalent to the level of compression you would get with "gzip -6" compression.

The beauty of piping all of these options together is that you can tune your pipeline to get the best results. You would have to experiment to see which gives you better performance*: enabling compression in the SSH connection, or enabling compression at the tar/rsync level.

Tar can use different compression options (gzip, bzip2, lzma, etc), and using the "-z" option with rsync will enable gzip compression. You could also add a separate compression/decompression program to both ends of the pipeline to tune the compression levels; here is an example of adding bzip2 compression, while explicitly disabling compression on the SSH connection:

tar cf - ~/Documents/projects/myproject/dist/ | bzip2 -c -9 | ssh -o Compression=no root@13.21.13.12 "bunzip2 -c | tar -C ~/public_html/ -xf -"

*To measure "performance", you need to decide what is most important for your application: CPU load, amount of bytes on the network, time required to perform the compression, etc. You can tune your pipeline and measure to see how changing the compression level and where in the pipeline the compression is applied affects these variables.

You will likely not improve performance by enabling compression in more than one spot in your pipeline.

  • Thanks +1.. might be useful for other users. for me rsync worked and I don't want to try out something else at this stage. – Mr_Green Jul 4 at 11:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.